Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X....X.....X.........X...X.X....X.X...X........X.....X....X] duration: 5.001117192s trace: https://jaeger.dev.storj.io/trace/77997aed52c8a82d FAIL (2025-12-30 09:21:34 UTC) orbiter warning: context deadline exceeded Duration: 5.00114291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.........X.X.X.X......X....X...X..XX..XX......X...X...X...] duration: 5.796515856s trace: https://jaeger.dev.storj.io/trace/24aac111015fad99 FAIL (2025-12-30 09:21:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ) context deadline exceeded ======LOGS====== 30 09:19:40.718173593 (593h50m40.589256648s) [upload=520596]: Starting upload Dec 30 09:19:40.718467532 (593h50m40.589550597s) [upload=520596]: Done starting upload Dec 30 09:19:40.718851944 (593h50m40.589935029s) [upload=520596]: Uploading segments... Dec 30 09:19:40.718917759 (593h50m40.590000804s) [upload=520596]: Got next segment. Inline: false Dec 30 09:19:40.718996540 (593h50m40.590079615s) [upload=520596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:19:40.719002832 (593h50m40.590085877s) [upload=520596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:19:40.962865837 (593h50m40.833948892s) [seg_pos={0 0}, upload=520596]: Begin upload segment... Dec 30 09:19:40.962926743 (593h50m40.834009788s) [seg_pos={0 0}, upload=520596]: Done begin upload segment. Dec 30 09:19:40.962965467 (593h50m40.834048512s) [noise=true , address=145.79.227.14:4838 , piece_id=6MVNA6CTX4NBYNWJOMYX5GKI4ZUEBGCJ5WRXXQ2FXLEVA7ZCAPRA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.966026425 (593h50m40.83710946s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.966831177 (593h50m40.837914223s) [noise=true , address=145.79.227.50:4373 , piece_id=KPVJOWE2OIHIJWW67NYJWHGKDSWHTXXOBIOEQ6PFAIOMSELBY5MA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.967351408 (593h50m40.838434443s) [noise=true , address=45.91.116.9:4262 , piece_id=ZU6JESEM4GYLY5XPNO75EFKDBA7CZ7LO73XQEDJEFMZVSSFKRP4A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.968020813 (593h50m40.839103848s) [noise=true , address=145.79.227.17:4972 , piece_id=VWJ5CTHGIJOBFJADRSRF62KJDOQBY72UNOFXJI3KZ3RX6FPP4WCQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.968537917 (593h50m40.839620962s) [noise=true , address=145.79.227.16:4909 , piece_id=ZYWDYODSND4Y7OMSAZUCZGVSULQKPGDS6K6ASAWWOTLMILCJTYAQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.970123497 (593h50m40.841206532s) [noise=true , address=77.90.31.11:4553 , piece_id=ANCQQ5RVJ5CWHPKF2TZZ4QYG4YILLPDSHMP6UI66FUZTYASUEA5A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.970948427 (593h50m40.842031472s) [noise=true , address=145.79.227.22:4510 , piece_id=ZSMJEW5IA3UCNYON3CLUKAFHWI5I4DYELBTNXOEUVVZSD2Z2ANWA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.971784379 (593h50m40.842867414s) [noise=true , address=145.79.227.7:4294 , piece_id=7ITAKWZCRET6UBYQQJ7LG5ECVGCIRGZDA4UGI7A74IGFV5TYYH4A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.973606640 (593h50m40.844689685s) [noise=true , address=45.91.116.8:4048 , piece_id=EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.974203025 (593h50m40.84528608s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.977280164 (593h50m40.848363229s) [noise=true , address=77.90.31.10:4519 , piece_id=JJJ4PNQBI4CS32UAYG3NCBI6BWLADY23FJKHH3VFAWK765RL63YA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.980643158 (593h50m40.851726233s) [noise=true , address=145.79.227.11:4729 , piece_id=YB5TCSURK6AKF44TG4JISCKAMV4VSZXPASS5BRNJGBYCOVQ2QCKQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.982929973 (593h50m40.854013008s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.986249824 (593h50m40.857332879s) [noise=true , address=145.79.227.23:4546 , piece_id=B7BQC76NEZ3DQ2IWKIOQHMAWHKYV6MZGR4LQ4TOY5V567NL7C6MQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.987769909 (593h50m40.858852944s) [noise=true , address=45.91.116.7:4010 , piece_id=3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:42.683052908 (593h50m42.554135954s) [upload=520596]: Next returned nil segment Dec 30 09:19:42.683066123 (593h50m42.554149158s) [upload=520596]: Waiting for error group managing segments... Dec 30 09:19:42.968722070 (593h50m42.839805105s) [noise=true , address=145.79.227.22:4510 , piece_id=ZSMJEW5IA3UCNYON3CLUKAFHWI5I4DYELBTNXOEUVVZSD2Z2ANWA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.970126844 (593h50m42.841209879s) [noise=true , address=145.79.227.7:4294 , piece_id=7ITAKWZCRET6UBYQQJ7LG5ECVGCIRGZDA4UGI7A74IGFV5TYYH4A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.975593574 (593h50m42.846676609s) [noise=true , address=145.79.227.50:4373 , piece_id=KPVJOWE2OIHIJWW67NYJWHGKDSWHTXXOBIOEQ6PFAIOMSELBY5MA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.977626043 (593h50m42.848709089s) [noise=true , address=145.79.227.17:4972 , piece_id=VWJ5CTHGIJOBFJADRSRF62KJDOQBY72UNOFXJI3KZ3RX6FPP4WCQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.982021304 (593h50m42.853104349s) [noise=true , address=145.79.227.14:4838 , piece_id=6MVNA6CTX4NBYNWJOMYX5GKI4ZUEBGCJ5WRXXQ2FXLEVA7ZCAPRA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.995968200 (593h50m42.867051235s) [noise=true , address=145.79.227.23:4546 , piece_id=B7BQC76NEZ3DQ2IWKIOQHMAWHKYV6MZGR4LQ4TOY5V567NL7C6MQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.048788970 (593h50m42.919872005s) [noise=true , address=145.79.227.16:4909 , piece_id=ZYWDYODSND4Y7OMSAZUCZGVSULQKPGDS6K6ASAWWOTLMILCJTYAQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.054054667 (593h50m42.925137712s) [noise=true , address=145.79.227.11:4729 , piece_id=YB5TCSURK6AKF44TG4JISCKAMV4VSZXPASS5BRNJGBYCOVQ2QCKQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.233935620 (593h50m43.105018665s) [noise=true , address=77.90.31.11:4553 , piece_id=ANCQQ5RVJ5CWHPKF2TZZ4QYG4YILLPDSHMP6UI66FUZTYASUEA5A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.239758017 (593h50m43.110841082s) [noise=true , address=77.90.31.10:4519 , piece_id=JJJ4PNQBI4CS32UAYG3NCBI6BWLADY23FJKHH3VFAWK765RL63YA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.269164662 (593h50m43.140247707s) [noise=true , address=45.91.116.8:4048 , piece_id=EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.300545266 (593h50m43.171628301s) [noise=true , address=45.91.116.7:4010 , piece_id=3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.471559184 (593h50m43.342642249s) [noise=true , address=45.91.116.9:4262 , piece_id=ZU6JESEM4GYLY5XPNO75EFKDBA7CZ7LO73XQEDJEFMZVSSFKRP4A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.471596135 (593h50m43.34267918s) [seg_pos={0 0}, upload=520596]: Segment reached optimal threshold of 13 pieces. Dec 30 09:19:43.472715827 (593h50m43.343798902s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection Dec 30 09:19:43.472730044 (593h50m43.343813089s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473337400 (593h50m43.344420455s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 09:19:43.473350455 (593h50m43.34443352s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473637612 (593h50m43.344720667s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 09:19:43.473648863 (593h50m43.344731898s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473678089 (593h50m43.344761134s) [seg_pos={0 0}, upload=520596]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:19:43.473786986 (593h50m43.344870032s) [upload=520596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:19:43.473838484 (593h50m43.344921529s) [upload=520596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:19:43.764215440 (593h50m43.635298495s) [upload=520596]: Done uploading segments... Dec 30 09:19:43.764237282 (593h50m43.635320327s) [upload=520596]: Upload finished. err: <nil> Duration: 5.796547327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.....................X..........................X.......] duration: 40.150317414s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:20:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 40.150349736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.........X.X.X.X......X....X...X..XX..XX......X...X...X..] duration: 5.994030398s trace: https://jaeger.dev.storj.io/trace/24aac111015fad99 FAIL (2025-12-30 09:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ) context deadline exceeded ======LOGS====== 30 09:19:40.718173593 (593h50m40.589256648s) [upload=520596]: Starting upload Dec 30 09:19:40.718467532 (593h50m40.589550597s) [upload=520596]: Done starting upload Dec 30 09:19:40.718851944 (593h50m40.589935029s) [upload=520596]: Uploading segments... Dec 30 09:19:40.718917759 (593h50m40.590000804s) [upload=520596]: Got next segment. Inline: false Dec 30 09:19:40.718996540 (593h50m40.590079615s) [upload=520596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:19:40.719002832 (593h50m40.590085877s) [upload=520596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:19:40.962865837 (593h50m40.833948892s) [seg_pos={0 0}, upload=520596]: Begin upload segment... Dec 30 09:19:40.962926743 (593h50m40.834009788s) [seg_pos={0 0}, upload=520596]: Done begin upload segment. Dec 30 09:19:40.962965467 (593h50m40.834048512s) [noise=true , address=145.79.227.14:4838 , piece_id=6MVNA6CTX4NBYNWJOMYX5GKI4ZUEBGCJ5WRXXQ2FXLEVA7ZCAPRA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.966026425 (593h50m40.83710946s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.966831177 (593h50m40.837914223s) [noise=true , address=145.79.227.50:4373 , piece_id=KPVJOWE2OIHIJWW67NYJWHGKDSWHTXXOBIOEQ6PFAIOMSELBY5MA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.967351408 (593h50m40.838434443s) [noise=true , address=45.91.116.9:4262 , piece_id=ZU6JESEM4GYLY5XPNO75EFKDBA7CZ7LO73XQEDJEFMZVSSFKRP4A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.968020813 (593h50m40.839103848s) [noise=true , address=145.79.227.17:4972 , piece_id=VWJ5CTHGIJOBFJADRSRF62KJDOQBY72UNOFXJI3KZ3RX6FPP4WCQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.968537917 (593h50m40.839620962s) [noise=true , address=145.79.227.16:4909 , piece_id=ZYWDYODSND4Y7OMSAZUCZGVSULQKPGDS6K6ASAWWOTLMILCJTYAQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.970123497 (593h50m40.841206532s) [noise=true , address=77.90.31.11:4553 , piece_id=ANCQQ5RVJ5CWHPKF2TZZ4QYG4YILLPDSHMP6UI66FUZTYASUEA5A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.970948427 (593h50m40.842031472s) [noise=true , address=145.79.227.22:4510 , piece_id=ZSMJEW5IA3UCNYON3CLUKAFHWI5I4DYELBTNXOEUVVZSD2Z2ANWA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.971784379 (593h50m40.842867414s) [noise=true , address=145.79.227.7:4294 , piece_id=7ITAKWZCRET6UBYQQJ7LG5ECVGCIRGZDA4UGI7A74IGFV5TYYH4A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.973606640 (593h50m40.844689685s) [noise=true , address=45.91.116.8:4048 , piece_id=EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.974203025 (593h50m40.84528608s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.977280164 (593h50m40.848363229s) [noise=true , address=77.90.31.10:4519 , piece_id=JJJ4PNQBI4CS32UAYG3NCBI6BWLADY23FJKHH3VFAWK765RL63YA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.980643158 (593h50m40.851726233s) [noise=true , address=145.79.227.11:4729 , piece_id=YB5TCSURK6AKF44TG4JISCKAMV4VSZXPASS5BRNJGBYCOVQ2QCKQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.982929973 (593h50m40.854013008s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.986249824 (593h50m40.857332879s) [noise=true , address=145.79.227.23:4546 , piece_id=B7BQC76NEZ3DQ2IWKIOQHMAWHKYV6MZGR4LQ4TOY5V567NL7C6MQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.987769909 (593h50m40.858852944s) [noise=true , address=45.91.116.7:4010 , piece_id=3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:42.683052908 (593h50m42.554135954s) [upload=520596]: Next returned nil segment Dec 30 09:19:42.683066123 (593h50m42.554149158s) [upload=520596]: Waiting for error group managing segments... Dec 30 09:19:42.968722070 (593h50m42.839805105s) [noise=true , address=145.79.227.22:4510 , piece_id=ZSMJEW5IA3UCNYON3CLUKAFHWI5I4DYELBTNXOEUVVZSD2Z2ANWA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.970126844 (593h50m42.841209879s) [noise=true , address=145.79.227.7:4294 , piece_id=7ITAKWZCRET6UBYQQJ7LG5ECVGCIRGZDA4UGI7A74IGFV5TYYH4A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.975593574 (593h50m42.846676609s) [noise=true , address=145.79.227.50:4373 , piece_id=KPVJOWE2OIHIJWW67NYJWHGKDSWHTXXOBIOEQ6PFAIOMSELBY5MA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.977626043 (593h50m42.848709089s) [noise=true , address=145.79.227.17:4972 , piece_id=VWJ5CTHGIJOBFJADRSRF62KJDOQBY72UNOFXJI3KZ3RX6FPP4WCQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.982021304 (593h50m42.853104349s) [noise=true , address=145.79.227.14:4838 , piece_id=6MVNA6CTX4NBYNWJOMYX5GKI4ZUEBGCJ5WRXXQ2FXLEVA7ZCAPRA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.995968200 (593h50m42.867051235s) [noise=true , address=145.79.227.23:4546 , piece_id=B7BQC76NEZ3DQ2IWKIOQHMAWHKYV6MZGR4LQ4TOY5V567NL7C6MQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.048788970 (593h50m42.919872005s) [noise=true , address=145.79.227.16:4909 , piece_id=ZYWDYODSND4Y7OMSAZUCZGVSULQKPGDS6K6ASAWWOTLMILCJTYAQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.054054667 (593h50m42.925137712s) [noise=true , address=145.79.227.11:4729 , piece_id=YB5TCSURK6AKF44TG4JISCKAMV4VSZXPASS5BRNJGBYCOVQ2QCKQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.233935620 (593h50m43.105018665s) [noise=true , address=77.90.31.11:4553 , piece_id=ANCQQ5RVJ5CWHPKF2TZZ4QYG4YILLPDSHMP6UI66FUZTYASUEA5A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.239758017 (593h50m43.110841082s) [noise=true , address=77.90.31.10:4519 , piece_id=JJJ4PNQBI4CS32UAYG3NCBI6BWLADY23FJKHH3VFAWK765RL63YA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.269164662 (593h50m43.140247707s) [noise=true , address=45.91.116.8:4048 , piece_id=EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.300545266 (593h50m43.171628301s) [noise=true , address=45.91.116.7:4010 , piece_id=3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.471559184 (593h50m43.342642249s) [noise=true , address=45.91.116.9:4262 , piece_id=ZU6JESEM4GYLY5XPNO75EFKDBA7CZ7LO73XQEDJEFMZVSSFKRP4A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.471596135 (593h50m43.34267918s) [seg_pos={0 0}, upload=520596]: Segment reached optimal threshold of 13 pieces. Dec 30 09:19:43.472715827 (593h50m43.343798902s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection Dec 30 09:19:43.472730044 (593h50m43.343813089s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473337400 (593h50m43.344420455s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 09:19:43.473350455 (593h50m43.34443352s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473637612 (593h50m43.344720667s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 09:19:43.473648863 (593h50m43.344731898s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473678089 (593h50m43.344761134s) [seg_pos={0 0}, upload=520596]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:19:43.473786986 (593h50m43.344870032s) [upload=520596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:19:43.473838484 (593h50m43.344921529s) [upload=520596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:19:43.764215440 (593h50m43.635298495s) [upload=520596]: Done uploading segments... Dec 30 09:19:43.764237282 (593h50m43.635320327s) [upload=520596]: Upload finished. err: <nil> Duration: 5.994056247s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..X....X.....X.........X...X.X....X.X...X........X.....X....] duration: 3.438664416s trace: https://jaeger.dev.storj.io/trace/731d29a169fda304 FAIL (2025-12-30 09:20:27 UTC) orbiter warning: context deadline exceeded Duration: 3.438694906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X.........X.X.X.X......X....X...X..XX..XX......X...X...X.] duration: 5.815269917s trace: https://jaeger.dev.storj.io/trace/24aac111015fad99 FAIL (2025-12-30 09:20:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ) context deadline exceeded ======LOGS====== 30 09:19:40.718173593 (593h50m40.589256648s) [upload=520596]: Starting upload Dec 30 09:19:40.718467532 (593h50m40.589550597s) [upload=520596]: Done starting upload Dec 30 09:19:40.718851944 (593h50m40.589935029s) [upload=520596]: Uploading segments... Dec 30 09:19:40.718917759 (593h50m40.590000804s) [upload=520596]: Got next segment. Inline: false Dec 30 09:19:40.718996540 (593h50m40.590079615s) [upload=520596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:19:40.719002832 (593h50m40.590085877s) [upload=520596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:19:40.962865837 (593h50m40.833948892s) [seg_pos={0 0}, upload=520596]: Begin upload segment... Dec 30 09:19:40.962926743 (593h50m40.834009788s) [seg_pos={0 0}, upload=520596]: Done begin upload segment. Dec 30 09:19:40.962965467 (593h50m40.834048512s) [noise=true , address=145.79.227.14:4838 , piece_id=6MVNA6CTX4NBYNWJOMYX5GKI4ZUEBGCJ5WRXXQ2FXLEVA7ZCAPRA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.966026425 (593h50m40.83710946s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.966831177 (593h50m40.837914223s) [noise=true , address=145.79.227.50:4373 , piece_id=KPVJOWE2OIHIJWW67NYJWHGKDSWHTXXOBIOEQ6PFAIOMSELBY5MA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.967351408 (593h50m40.838434443s) [noise=true , address=45.91.116.9:4262 , piece_id=ZU6JESEM4GYLY5XPNO75EFKDBA7CZ7LO73XQEDJEFMZVSSFKRP4A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.968020813 (593h50m40.839103848s) [noise=true , address=145.79.227.17:4972 , piece_id=VWJ5CTHGIJOBFJADRSRF62KJDOQBY72UNOFXJI3KZ3RX6FPP4WCQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.968537917 (593h50m40.839620962s) [noise=true , address=145.79.227.16:4909 , piece_id=ZYWDYODSND4Y7OMSAZUCZGVSULQKPGDS6K6ASAWWOTLMILCJTYAQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.970123497 (593h50m40.841206532s) [noise=true , address=77.90.31.11:4553 , piece_id=ANCQQ5RVJ5CWHPKF2TZZ4QYG4YILLPDSHMP6UI66FUZTYASUEA5A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.970948427 (593h50m40.842031472s) [noise=true , address=145.79.227.22:4510 , piece_id=ZSMJEW5IA3UCNYON3CLUKAFHWI5I4DYELBTNXOEUVVZSD2Z2ANWA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.971784379 (593h50m40.842867414s) [noise=true , address=145.79.227.7:4294 , piece_id=7ITAKWZCRET6UBYQQJ7LG5ECVGCIRGZDA4UGI7A74IGFV5TYYH4A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.973606640 (593h50m40.844689685s) [noise=true , address=45.91.116.8:4048 , piece_id=EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.974203025 (593h50m40.84528608s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.977280164 (593h50m40.848363229s) [noise=true , address=77.90.31.10:4519 , piece_id=JJJ4PNQBI4CS32UAYG3NCBI6BWLADY23FJKHH3VFAWK765RL63YA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.980643158 (593h50m40.851726233s) [noise=true , address=145.79.227.11:4729 , piece_id=YB5TCSURK6AKF44TG4JISCKAMV4VSZXPASS5BRNJGBYCOVQ2QCKQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.982929973 (593h50m40.854013008s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.986249824 (593h50m40.857332879s) [noise=true , address=145.79.227.23:4546 , piece_id=B7BQC76NEZ3DQ2IWKIOQHMAWHKYV6MZGR4LQ4TOY5V567NL7C6MQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.987769909 (593h50m40.858852944s) [noise=true , address=45.91.116.7:4010 , piece_id=3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:42.683052908 (593h50m42.554135954s) [upload=520596]: Next returned nil segment Dec 30 09:19:42.683066123 (593h50m42.554149158s) [upload=520596]: Waiting for error group managing segments... Dec 30 09:19:42.968722070 (593h50m42.839805105s) [noise=true , address=145.79.227.22:4510 , piece_id=ZSMJEW5IA3UCNYON3CLUKAFHWI5I4DYELBTNXOEUVVZSD2Z2ANWA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.970126844 (593h50m42.841209879s) [noise=true , address=145.79.227.7:4294 , piece_id=7ITAKWZCRET6UBYQQJ7LG5ECVGCIRGZDA4UGI7A74IGFV5TYYH4A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.975593574 (593h50m42.846676609s) [noise=true , address=145.79.227.50:4373 , piece_id=KPVJOWE2OIHIJWW67NYJWHGKDSWHTXXOBIOEQ6PFAIOMSELBY5MA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.977626043 (593h50m42.848709089s) [noise=true , address=145.79.227.17:4972 , piece_id=VWJ5CTHGIJOBFJADRSRF62KJDOQBY72UNOFXJI3KZ3RX6FPP4WCQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.982021304 (593h50m42.853104349s) [noise=true , address=145.79.227.14:4838 , piece_id=6MVNA6CTX4NBYNWJOMYX5GKI4ZUEBGCJ5WRXXQ2FXLEVA7ZCAPRA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.995968200 (593h50m42.867051235s) [noise=true , address=145.79.227.23:4546 , piece_id=B7BQC76NEZ3DQ2IWKIOQHMAWHKYV6MZGR4LQ4TOY5V567NL7C6MQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.048788970 (593h50m42.919872005s) [noise=true , address=145.79.227.16:4909 , piece_id=ZYWDYODSND4Y7OMSAZUCZGVSULQKPGDS6K6ASAWWOTLMILCJTYAQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.054054667 (593h50m42.925137712s) [noise=true , address=145.79.227.11:4729 , piece_id=YB5TCSURK6AKF44TG4JISCKAMV4VSZXPASS5BRNJGBYCOVQ2QCKQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.233935620 (593h50m43.105018665s) [noise=true , address=77.90.31.11:4553 , piece_id=ANCQQ5RVJ5CWHPKF2TZZ4QYG4YILLPDSHMP6UI66FUZTYASUEA5A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.239758017 (593h50m43.110841082s) [noise=true , address=77.90.31.10:4519 , piece_id=JJJ4PNQBI4CS32UAYG3NCBI6BWLADY23FJKHH3VFAWK765RL63YA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.269164662 (593h50m43.140247707s) [noise=true , address=45.91.116.8:4048 , piece_id=EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.300545266 (593h50m43.171628301s) [noise=true , address=45.91.116.7:4010 , piece_id=3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.471559184 (593h50m43.342642249s) [noise=true , address=45.91.116.9:4262 , piece_id=ZU6JESEM4GYLY5XPNO75EFKDBA7CZ7LO73XQEDJEFMZVSSFKRP4A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.471596135 (593h50m43.34267918s) [seg_pos={0 0}, upload=520596]: Segment reached optimal threshold of 13 pieces. Dec 30 09:19:43.472715827 (593h50m43.343798902s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection Dec 30 09:19:43.472730044 (593h50m43.343813089s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473337400 (593h50m43.344420455s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 09:19:43.473350455 (593h50m43.34443352s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473637612 (593h50m43.344720667s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 09:19:43.473648863 (593h50m43.344731898s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473678089 (593h50m43.344761134s) [seg_pos={0 0}, upload=520596]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:19:43.473786986 (593h50m43.344870032s) [upload=520596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:19:43.473838484 (593h50m43.344921529s) [upload=520596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:19:43.764215440 (593h50m43.635298495s) [upload=520596]: Done uploading segments... Dec 30 09:19:43.764237282 (593h50m43.635320327s) [upload=520596]: Upload finished. err: <nil> Duration: 5.815288713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X.........X.X.X.X......X....X...X..XX..XX......X...X...X] duration: 30.004150055s trace: https://jaeger.dev.storj.io/trace/24aac111015fad99 FAIL (2025-12-30 09:20:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: 3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ) context deadline exceeded ======LOGS====== 30 09:19:40.718173593 (593h50m40.589256648s) [upload=520596]: Starting upload Dec 30 09:19:40.718467532 (593h50m40.589550597s) [upload=520596]: Done starting upload Dec 30 09:19:40.718851944 (593h50m40.589935029s) [upload=520596]: Uploading segments... Dec 30 09:19:40.718917759 (593h50m40.590000804s) [upload=520596]: Got next segment. Inline: false Dec 30 09:19:40.718996540 (593h50m40.590079615s) [upload=520596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:19:40.719002832 (593h50m40.590085877s) [upload=520596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:19:40.962865837 (593h50m40.833948892s) [seg_pos={0 0}, upload=520596]: Begin upload segment... Dec 30 09:19:40.962926743 (593h50m40.834009788s) [seg_pos={0 0}, upload=520596]: Done begin upload segment. Dec 30 09:19:40.962965467 (593h50m40.834048512s) [noise=true , address=145.79.227.14:4838 , piece_id=6MVNA6CTX4NBYNWJOMYX5GKI4ZUEBGCJ5WRXXQ2FXLEVA7ZCAPRA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.966026425 (593h50m40.83710946s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.966831177 (593h50m40.837914223s) [noise=true , address=145.79.227.50:4373 , piece_id=KPVJOWE2OIHIJWW67NYJWHGKDSWHTXXOBIOEQ6PFAIOMSELBY5MA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.967351408 (593h50m40.838434443s) [noise=true , address=45.91.116.9:4262 , piece_id=ZU6JESEM4GYLY5XPNO75EFKDBA7CZ7LO73XQEDJEFMZVSSFKRP4A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.968020813 (593h50m40.839103848s) [noise=true , address=145.79.227.17:4972 , piece_id=VWJ5CTHGIJOBFJADRSRF62KJDOQBY72UNOFXJI3KZ3RX6FPP4WCQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.968537917 (593h50m40.839620962s) [noise=true , address=145.79.227.16:4909 , piece_id=ZYWDYODSND4Y7OMSAZUCZGVSULQKPGDS6K6ASAWWOTLMILCJTYAQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.970123497 (593h50m40.841206532s) [noise=true , address=77.90.31.11:4553 , piece_id=ANCQQ5RVJ5CWHPKF2TZZ4QYG4YILLPDSHMP6UI66FUZTYASUEA5A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.970948427 (593h50m40.842031472s) [noise=true , address=145.79.227.22:4510 , piece_id=ZSMJEW5IA3UCNYON3CLUKAFHWI5I4DYELBTNXOEUVVZSD2Z2ANWA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.971784379 (593h50m40.842867414s) [noise=true , address=145.79.227.7:4294 , piece_id=7ITAKWZCRET6UBYQQJ7LG5ECVGCIRGZDA4UGI7A74IGFV5TYYH4A, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.973606640 (593h50m40.844689685s) [noise=true , address=45.91.116.8:4048 , piece_id=EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.974203025 (593h50m40.84528608s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.977280164 (593h50m40.848363229s) [noise=true , address=77.90.31.10:4519 , piece_id=JJJ4PNQBI4CS32UAYG3NCBI6BWLADY23FJKHH3VFAWK765RL63YA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.980643158 (593h50m40.851726233s) [noise=true , address=145.79.227.11:4729 , piece_id=YB5TCSURK6AKF44TG4JISCKAMV4VSZXPASS5BRNJGBYCOVQ2QCKQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.982929973 (593h50m40.854013008s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.986249824 (593h50m40.857332879s) [noise=true , address=145.79.227.23:4546 , piece_id=B7BQC76NEZ3DQ2IWKIOQHMAWHKYV6MZGR4LQ4TOY5V567NL7C6MQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:40.987769909 (593h50m40.858852944s) [noise=true , address=45.91.116.7:4010 , piece_id=3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ, seg_pos={0 0}, upload=520596]: Uploading piece... Dec 30 09:19:42.683052908 (593h50m42.554135954s) [upload=520596]: Next returned nil segment Dec 30 09:19:42.683066123 (593h50m42.554149158s) [upload=520596]: Waiting for error group managing segments... Dec 30 09:19:42.968722070 (593h50m42.839805105s) [noise=true , address=145.79.227.22:4510 , piece_id=ZSMJEW5IA3UCNYON3CLUKAFHWI5I4DYELBTNXOEUVVZSD2Z2ANWA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.970126844 (593h50m42.841209879s) [noise=true , address=145.79.227.7:4294 , piece_id=7ITAKWZCRET6UBYQQJ7LG5ECVGCIRGZDA4UGI7A74IGFV5TYYH4A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.975593574 (593h50m42.846676609s) [noise=true , address=145.79.227.50:4373 , piece_id=KPVJOWE2OIHIJWW67NYJWHGKDSWHTXXOBIOEQ6PFAIOMSELBY5MA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.977626043 (593h50m42.848709089s) [noise=true , address=145.79.227.17:4972 , piece_id=VWJ5CTHGIJOBFJADRSRF62KJDOQBY72UNOFXJI3KZ3RX6FPP4WCQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.982021304 (593h50m42.853104349s) [noise=true , address=145.79.227.14:4838 , piece_id=6MVNA6CTX4NBYNWJOMYX5GKI4ZUEBGCJ5WRXXQ2FXLEVA7ZCAPRA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:42.995968200 (593h50m42.867051235s) [noise=true , address=145.79.227.23:4546 , piece_id=B7BQC76NEZ3DQ2IWKIOQHMAWHKYV6MZGR4LQ4TOY5V567NL7C6MQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.048788970 (593h50m42.919872005s) [noise=true , address=145.79.227.16:4909 , piece_id=ZYWDYODSND4Y7OMSAZUCZGVSULQKPGDS6K6ASAWWOTLMILCJTYAQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.054054667 (593h50m42.925137712s) [noise=true , address=145.79.227.11:4729 , piece_id=YB5TCSURK6AKF44TG4JISCKAMV4VSZXPASS5BRNJGBYCOVQ2QCKQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.233935620 (593h50m43.105018665s) [noise=true , address=77.90.31.11:4553 , piece_id=ANCQQ5RVJ5CWHPKF2TZZ4QYG4YILLPDSHMP6UI66FUZTYASUEA5A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.239758017 (593h50m43.110841082s) [noise=true , address=77.90.31.10:4519 , piece_id=JJJ4PNQBI4CS32UAYG3NCBI6BWLADY23FJKHH3VFAWK765RL63YA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.269164662 (593h50m43.140247707s) [noise=true , address=45.91.116.8:4048 , piece_id=EIF6N7CWHXSM6NL6AEZY5HWURSJENBJIXOR7ZFPJ6U6INJIPXBVQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.300545266 (593h50m43.171628301s) [noise=true , address=45.91.116.7:4010 , piece_id=3BQ5JL2322LGDZJ7SEYGHHAXWA3A5RWJLVH6S7XLAUNLVKUF6UBQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.471559184 (593h50m43.342642249s) [noise=true , address=45.91.116.9:4262 , piece_id=ZU6JESEM4GYLY5XPNO75EFKDBA7CZ7LO73XQEDJEFMZVSSFKRP4A, seg_pos={0 0}, upload=520596]: Done uploading piece. err: <nil> Dec 30 09:19:43.471596135 (593h50m43.34267918s) [seg_pos={0 0}, upload=520596]: Segment reached optimal threshold of 13 pieces. Dec 30 09:19:43.472715827 (593h50m43.343798902s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54048->77.90.31.16:4735: use of closed network connection Dec 30 09:19:43.472730044 (593h50m43.343813089s) [noise=true , address=77.90.31.16:4735 , piece_id=GXWXXOQWPVZNGJJF32DGLOQQZLWNWSI7MFZNPSV7GGTKTOAZAH6Q, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473337400 (593h50m43.344420455s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 09:19:43.473350455 (593h50m43.34443352s) [noise=true , address=45.91.116.4:4937 , piece_id=OSJCUUZRJBKKL3LVQ4CJMJAL2P4ONUG6CSW4VE27XKWVF4IIX5RA, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473637612 (593h50m43.344720667s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 09:19:43.473648863 (593h50m43.344731898s) [noise=true , address=77.90.31.15:4697 , piece_id=46GTRCABFARUAUFSUH7OQJFUZG4PY7Q3EEWUDKHJUEXCRTEBMYBQ, seg_pos={0 0}, upload=520596]: Long tail cancellation due to optimal threshold Dec 30 09:19:43.473678089 (593h50m43.344761134s) [seg_pos={0 0}, upload=520596]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:19:43.473786986 (593h50m43.344870032s) [upload=520596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:19:43.473838484 (593h50m43.344921529s) [upload=520596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:19:43.764215440 (593h50m43.635298495s) [upload=520596]: Done uploading segments... Dec 30 09:19:43.764237282 (593h50m43.635320327s) [upload=520596]: Upload finished. err: <nil> Duration: 30.004185872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.....................X..........................X......] duration: 41.683142315s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:20:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 41.683165349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X....X.....X.........X...X.X....X.X...X........X.....X...] duration: 3.529296295s trace: https://jaeger.dev.storj.io/trace/731d29a169fda304 FAIL (2025-12-30 09:19:25 UTC) orbiter warning: context deadline exceeded Duration: 3.529322535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X.X.........X.X.X.X......X....X...X..XX..XX......X...X...] duration: 5.684390801s trace: https://jaeger.dev.storj.io/trace/11890a965976933 FAIL (2025-12-30 09:19:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A) context deadline exceeded; piecestore: (Node ID: 1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, Piece ID: VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA) context deadline exceeded; piecestore: (Node ID: 17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, Piece ID: WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA) context deadline exceeded ======LOGS====== 7:32.333269072 (593h48m32.204352117s) [upload=520564]: Starting upload Dec 30 09:17:32.333352361 (593h48m32.204435396s) [upload=520564]: Done starting upload Dec 30 09:17:32.333446972 (593h48m32.204530007s) [upload=520564]: Uploading segments... Dec 30 09:17:32.333686959 (593h48m32.204770004s) [upload=520564]: Got next segment. Inline: false Dec 30 09:17:32.333715523 (593h48m32.204798568s) [upload=520564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:17:32.333722256 (593h48m32.204805301s) [upload=520564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:17:32.581711087 (593h48m32.452794132s) [seg_pos={0 0}, upload=520564]: Begin upload segment... Dec 30 09:17:32.581787193 (593h48m32.452870238s) [seg_pos={0 0}, upload=520564]: Done begin upload segment. Dec 30 09:17:32.581816698 (593h48m32.452899743s) [noise=true , address=145.79.227.15:4876 , piece_id=G3OJRP6MW4LFWV4JNGSP7JSXZDUBDSAOHVJHGMTAYZGIBOUFJHUQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.581966834 (593h48m32.453049869s) [noise=true , address=145.79.227.36:4518 , piece_id=WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.582369851 (593h48m32.453452896s) [noise=true , address=145.79.227.50:4374 , piece_id=KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.583188921 (593h48m32.454271966s) [noise=true , address=145.79.227.31:4799 , piece_id=GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.583605244 (593h48m32.454688289s) [noise=true , address=145.79.227.32:4834 , piece_id=LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584119574 (593h48m32.455202609s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584506260 (593h48m32.455589295s) [noise=true , address=45.91.116.9:4262 , piece_id=4VSX2SZQD62B3T5KSQKQOE52DNYBC27S7FLH6EHZ6R2MPMW3A74Q, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584880643 (593h48m32.455963678s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.585259754 (593h48m32.456342789s) [noise=true , address=145.79.227.14:4839 , piece_id=C55BUV5NDF3CEWZ3RNHAIDYXLMIVXYCDNWTXF744473KCBG3DAVQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.586159818 (593h48m32.457242853s) [noise=true , address=145.79.227.45:4229 , piece_id=SR3I6K7YMVIZXRZOOUE7TSQGZYHGSNF2EWHNUHGSHZV4HQPUOZHQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.588541834 (593h48m32.45962488s) [noise=true , address=145.79.227.11:4729 , piece_id=VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.591956266 (593h48m32.463039311s) [noise=true , address=77.90.31.18:4808 , piece_id=MU5OMFVVMLWQEYGF3YZFKFIFKZMKHCS62ZKQ3YMGWLT57DSOOR2A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.593366611 (593h48m32.464449656s) [noise=true , address=145.79.227.9:4365 , piece_id=FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.594992598 (593h48m32.466075643s) [noise=true , address=77.90.31.17:4773 , piece_id=TYR46DEEB6MRGHVCJG4RGUMJGGNSPKGBKZADYO4OIP35JE76ZHVQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.596337158 (593h48m32.467420203s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.597969637 (593h48m32.469052682s) [noise=true , address=145.79.227.46:4603 , piece_id=3753VEM45GQH6ZM3HQRPI4YNQKQGUX7HUXGDF2IAZH7SZLCXNPTQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:34.496610974 (593h48m34.367694009s) [upload=520564]: Next returned nil segment Dec 30 09:17:34.496641451 (593h48m34.367724486s) [upload=520564]: Waiting for error group managing segments... Dec 30 09:17:34.820502297 (593h48m34.691585332s) [noise=true , address=145.79.227.31:4799 , piece_id=GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.827819970 (593h48m34.698903016s) [noise=true , address=145.79.227.32:4834 , piece_id=LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.828989467 (593h48m34.700072502s) [noise=true , address=77.90.31.18:4808 , piece_id=MU5OMFVVMLWQEYGF3YZFKFIFKZMKHCS62ZKQ3YMGWLT57DSOOR2A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.849552623 (593h48m34.720635668s) [noise=true , address=145.79.227.11:4729 , piece_id=VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.870010248 (593h48m34.741093293s) [noise=true , address=145.79.227.50:4374 , piece_id=KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.887563172 (593h48m34.758646207s) [noise=true , address=45.91.116.9:4262 , piece_id=4VSX2SZQD62B3T5KSQKQOE52DNYBC27S7FLH6EHZ6R2MPMW3A74Q, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.981545660 (593h48m34.852628695s) [noise=true , address=77.90.31.17:4773 , piece_id=TYR46DEEB6MRGHVCJG4RGUMJGGNSPKGBKZADYO4OIP35JE76ZHVQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.484542950 (593h48m35.355625985s) [noise=true , address=145.79.227.45:4229 , piece_id=SR3I6K7YMVIZXRZOOUE7TSQGZYHGSNF2EWHNUHGSHZV4HQPUOZHQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.507515103 (593h48m35.378598148s) [noise=true , address=145.79.227.9:4365 , piece_id=FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.573332239 (593h48m35.444415274s) [noise=true , address=145.79.227.36:4518 , piece_id=WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.083563394 (593h48m35.954646429s) [noise=true , address=145.79.227.14:4839 , piece_id=C55BUV5NDF3CEWZ3RNHAIDYXLMIVXYCDNWTXF744473KCBG3DAVQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.222079772 (593h48m36.093162807s) [noise=true , address=145.79.227.15:4876 , piece_id=G3OJRP6MW4LFWV4JNGSP7JSXZDUBDSAOHVJHGMTAYZGIBOUFJHUQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.310181082 (593h48m36.181264128s) [noise=true , address=145.79.227.46:4603 , piece_id=3753VEM45GQH6ZM3HQRPI4YNQKQGUX7HUXGDF2IAZH7SZLCXNPTQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.310200128 (593h48m36.181283163s) [seg_pos={0 0}, upload=520564]: Segment reached optimal threshold of 13 pieces. Dec 30 09:17:36.310360454 (593h48m36.181443489s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 30 09:17:36.310366275 (593h48m36.18144931s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310474010 (593h48m36.181557045s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 30 09:17:36.310482396 (593h48m36.181565431s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310557469 (593h48m36.181640504s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection Dec 30 09:17:36.310563931 (593h48m36.181646977s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310577156 (593h48m36.181660201s) [seg_pos={0 0}, upload=520564]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:17:36.310639726 (593h48m36.181722761s) [upload=520564]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:17:36.310643683 (593h48m36.181726718s) [upload=520564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:17:36.607104834 (593h48m36.478187869s) [upload=520564]: Done uploading segments... Dec 30 09:17:36.607124372 (593h48m36.478207407s) [upload=520564]: Upload finished. err: <nil> Duration: 5.684417353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.....................X..........................X.....] duration: 26.596125519s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:18:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 26.596153783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X...X..] duration: 6.042490972s trace: https://jaeger.dev.storj.io/trace/11890a965976933 FAIL (2025-12-30 09:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A) context deadline exceeded; piecestore: (Node ID: 1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, Piece ID: VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA) context deadline exceeded; piecestore: (Node ID: 17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, Piece ID: WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA) context deadline exceeded ======LOGS====== 7:32.333269072 (593h48m32.204352117s) [upload=520564]: Starting upload Dec 30 09:17:32.333352361 (593h48m32.204435396s) [upload=520564]: Done starting upload Dec 30 09:17:32.333446972 (593h48m32.204530007s) [upload=520564]: Uploading segments... Dec 30 09:17:32.333686959 (593h48m32.204770004s) [upload=520564]: Got next segment. Inline: false Dec 30 09:17:32.333715523 (593h48m32.204798568s) [upload=520564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:17:32.333722256 (593h48m32.204805301s) [upload=520564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:17:32.581711087 (593h48m32.452794132s) [seg_pos={0 0}, upload=520564]: Begin upload segment... Dec 30 09:17:32.581787193 (593h48m32.452870238s) [seg_pos={0 0}, upload=520564]: Done begin upload segment. Dec 30 09:17:32.581816698 (593h48m32.452899743s) [noise=true , address=145.79.227.15:4876 , piece_id=G3OJRP6MW4LFWV4JNGSP7JSXZDUBDSAOHVJHGMTAYZGIBOUFJHUQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.581966834 (593h48m32.453049869s) [noise=true , address=145.79.227.36:4518 , piece_id=WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.582369851 (593h48m32.453452896s) [noise=true , address=145.79.227.50:4374 , piece_id=KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.583188921 (593h48m32.454271966s) [noise=true , address=145.79.227.31:4799 , piece_id=GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.583605244 (593h48m32.454688289s) [noise=true , address=145.79.227.32:4834 , piece_id=LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584119574 (593h48m32.455202609s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584506260 (593h48m32.455589295s) [noise=true , address=45.91.116.9:4262 , piece_id=4VSX2SZQD62B3T5KSQKQOE52DNYBC27S7FLH6EHZ6R2MPMW3A74Q, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584880643 (593h48m32.455963678s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.585259754 (593h48m32.456342789s) [noise=true , address=145.79.227.14:4839 , piece_id=C55BUV5NDF3CEWZ3RNHAIDYXLMIVXYCDNWTXF744473KCBG3DAVQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.586159818 (593h48m32.457242853s) [noise=true , address=145.79.227.45:4229 , piece_id=SR3I6K7YMVIZXRZOOUE7TSQGZYHGSNF2EWHNUHGSHZV4HQPUOZHQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.588541834 (593h48m32.45962488s) [noise=true , address=145.79.227.11:4729 , piece_id=VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.591956266 (593h48m32.463039311s) [noise=true , address=77.90.31.18:4808 , piece_id=MU5OMFVVMLWQEYGF3YZFKFIFKZMKHCS62ZKQ3YMGWLT57DSOOR2A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.593366611 (593h48m32.464449656s) [noise=true , address=145.79.227.9:4365 , piece_id=FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.594992598 (593h48m32.466075643s) [noise=true , address=77.90.31.17:4773 , piece_id=TYR46DEEB6MRGHVCJG4RGUMJGGNSPKGBKZADYO4OIP35JE76ZHVQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.596337158 (593h48m32.467420203s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.597969637 (593h48m32.469052682s) [noise=true , address=145.79.227.46:4603 , piece_id=3753VEM45GQH6ZM3HQRPI4YNQKQGUX7HUXGDF2IAZH7SZLCXNPTQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:34.496610974 (593h48m34.367694009s) [upload=520564]: Next returned nil segment Dec 30 09:17:34.496641451 (593h48m34.367724486s) [upload=520564]: Waiting for error group managing segments... Dec 30 09:17:34.820502297 (593h48m34.691585332s) [noise=true , address=145.79.227.31:4799 , piece_id=GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.827819970 (593h48m34.698903016s) [noise=true , address=145.79.227.32:4834 , piece_id=LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.828989467 (593h48m34.700072502s) [noise=true , address=77.90.31.18:4808 , piece_id=MU5OMFVVMLWQEYGF3YZFKFIFKZMKHCS62ZKQ3YMGWLT57DSOOR2A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.849552623 (593h48m34.720635668s) [noise=true , address=145.79.227.11:4729 , piece_id=VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.870010248 (593h48m34.741093293s) [noise=true , address=145.79.227.50:4374 , piece_id=KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.887563172 (593h48m34.758646207s) [noise=true , address=45.91.116.9:4262 , piece_id=4VSX2SZQD62B3T5KSQKQOE52DNYBC27S7FLH6EHZ6R2MPMW3A74Q, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.981545660 (593h48m34.852628695s) [noise=true , address=77.90.31.17:4773 , piece_id=TYR46DEEB6MRGHVCJG4RGUMJGGNSPKGBKZADYO4OIP35JE76ZHVQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.484542950 (593h48m35.355625985s) [noise=true , address=145.79.227.45:4229 , piece_id=SR3I6K7YMVIZXRZOOUE7TSQGZYHGSNF2EWHNUHGSHZV4HQPUOZHQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.507515103 (593h48m35.378598148s) [noise=true , address=145.79.227.9:4365 , piece_id=FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.573332239 (593h48m35.444415274s) [noise=true , address=145.79.227.36:4518 , piece_id=WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.083563394 (593h48m35.954646429s) [noise=true , address=145.79.227.14:4839 , piece_id=C55BUV5NDF3CEWZ3RNHAIDYXLMIVXYCDNWTXF744473KCBG3DAVQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.222079772 (593h48m36.093162807s) [noise=true , address=145.79.227.15:4876 , piece_id=G3OJRP6MW4LFWV4JNGSP7JSXZDUBDSAOHVJHGMTAYZGIBOUFJHUQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.310181082 (593h48m36.181264128s) [noise=true , address=145.79.227.46:4603 , piece_id=3753VEM45GQH6ZM3HQRPI4YNQKQGUX7HUXGDF2IAZH7SZLCXNPTQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.310200128 (593h48m36.181283163s) [seg_pos={0 0}, upload=520564]: Segment reached optimal threshold of 13 pieces. Dec 30 09:17:36.310360454 (593h48m36.181443489s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 30 09:17:36.310366275 (593h48m36.18144931s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310474010 (593h48m36.181557045s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 30 09:17:36.310482396 (593h48m36.181565431s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310557469 (593h48m36.181640504s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection Dec 30 09:17:36.310563931 (593h48m36.181646977s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310577156 (593h48m36.181660201s) [seg_pos={0 0}, upload=520564]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:17:36.310639726 (593h48m36.181722761s) [upload=520564]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:17:36.310643683 (593h48m36.181726718s) [upload=520564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:17:36.607104834 (593h48m36.478187869s) [upload=520564]: Done uploading segments... Dec 30 09:17:36.607124372 (593h48m36.478207407s) [upload=520564]: Upload finished. err: <nil> Duration: 6.042541869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X...X.] duration: 6.47010453s trace: https://jaeger.dev.storj.io/trace/11890a965976933 FAIL (2025-12-30 09:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A) context deadline exceeded; piecestore: (Node ID: 1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, Piece ID: VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA) context deadline exceeded; piecestore: (Node ID: 17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, Piece ID: WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA) context deadline exceeded ======LOGS====== 7:32.333269072 (593h48m32.204352117s) [upload=520564]: Starting upload Dec 30 09:17:32.333352361 (593h48m32.204435396s) [upload=520564]: Done starting upload Dec 30 09:17:32.333446972 (593h48m32.204530007s) [upload=520564]: Uploading segments... Dec 30 09:17:32.333686959 (593h48m32.204770004s) [upload=520564]: Got next segment. Inline: false Dec 30 09:17:32.333715523 (593h48m32.204798568s) [upload=520564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:17:32.333722256 (593h48m32.204805301s) [upload=520564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:17:32.581711087 (593h48m32.452794132s) [seg_pos={0 0}, upload=520564]: Begin upload segment... Dec 30 09:17:32.581787193 (593h48m32.452870238s) [seg_pos={0 0}, upload=520564]: Done begin upload segment. Dec 30 09:17:32.581816698 (593h48m32.452899743s) [noise=true , address=145.79.227.15:4876 , piece_id=G3OJRP6MW4LFWV4JNGSP7JSXZDUBDSAOHVJHGMTAYZGIBOUFJHUQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.581966834 (593h48m32.453049869s) [noise=true , address=145.79.227.36:4518 , piece_id=WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.582369851 (593h48m32.453452896s) [noise=true , address=145.79.227.50:4374 , piece_id=KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.583188921 (593h48m32.454271966s) [noise=true , address=145.79.227.31:4799 , piece_id=GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.583605244 (593h48m32.454688289s) [noise=true , address=145.79.227.32:4834 , piece_id=LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584119574 (593h48m32.455202609s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584506260 (593h48m32.455589295s) [noise=true , address=45.91.116.9:4262 , piece_id=4VSX2SZQD62B3T5KSQKQOE52DNYBC27S7FLH6EHZ6R2MPMW3A74Q, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584880643 (593h48m32.455963678s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.585259754 (593h48m32.456342789s) [noise=true , address=145.79.227.14:4839 , piece_id=C55BUV5NDF3CEWZ3RNHAIDYXLMIVXYCDNWTXF744473KCBG3DAVQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.586159818 (593h48m32.457242853s) [noise=true , address=145.79.227.45:4229 , piece_id=SR3I6K7YMVIZXRZOOUE7TSQGZYHGSNF2EWHNUHGSHZV4HQPUOZHQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.588541834 (593h48m32.45962488s) [noise=true , address=145.79.227.11:4729 , piece_id=VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.591956266 (593h48m32.463039311s) [noise=true , address=77.90.31.18:4808 , piece_id=MU5OMFVVMLWQEYGF3YZFKFIFKZMKHCS62ZKQ3YMGWLT57DSOOR2A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.593366611 (593h48m32.464449656s) [noise=true , address=145.79.227.9:4365 , piece_id=FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.594992598 (593h48m32.466075643s) [noise=true , address=77.90.31.17:4773 , piece_id=TYR46DEEB6MRGHVCJG4RGUMJGGNSPKGBKZADYO4OIP35JE76ZHVQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.596337158 (593h48m32.467420203s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.597969637 (593h48m32.469052682s) [noise=true , address=145.79.227.46:4603 , piece_id=3753VEM45GQH6ZM3HQRPI4YNQKQGUX7HUXGDF2IAZH7SZLCXNPTQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:34.496610974 (593h48m34.367694009s) [upload=520564]: Next returned nil segment Dec 30 09:17:34.496641451 (593h48m34.367724486s) [upload=520564]: Waiting for error group managing segments... Dec 30 09:17:34.820502297 (593h48m34.691585332s) [noise=true , address=145.79.227.31:4799 , piece_id=GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.827819970 (593h48m34.698903016s) [noise=true , address=145.79.227.32:4834 , piece_id=LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.828989467 (593h48m34.700072502s) [noise=true , address=77.90.31.18:4808 , piece_id=MU5OMFVVMLWQEYGF3YZFKFIFKZMKHCS62ZKQ3YMGWLT57DSOOR2A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.849552623 (593h48m34.720635668s) [noise=true , address=145.79.227.11:4729 , piece_id=VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.870010248 (593h48m34.741093293s) [noise=true , address=145.79.227.50:4374 , piece_id=KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.887563172 (593h48m34.758646207s) [noise=true , address=45.91.116.9:4262 , piece_id=4VSX2SZQD62B3T5KSQKQOE52DNYBC27S7FLH6EHZ6R2MPMW3A74Q, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.981545660 (593h48m34.852628695s) [noise=true , address=77.90.31.17:4773 , piece_id=TYR46DEEB6MRGHVCJG4RGUMJGGNSPKGBKZADYO4OIP35JE76ZHVQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.484542950 (593h48m35.355625985s) [noise=true , address=145.79.227.45:4229 , piece_id=SR3I6K7YMVIZXRZOOUE7TSQGZYHGSNF2EWHNUHGSHZV4HQPUOZHQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.507515103 (593h48m35.378598148s) [noise=true , address=145.79.227.9:4365 , piece_id=FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.573332239 (593h48m35.444415274s) [noise=true , address=145.79.227.36:4518 , piece_id=WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.083563394 (593h48m35.954646429s) [noise=true , address=145.79.227.14:4839 , piece_id=C55BUV5NDF3CEWZ3RNHAIDYXLMIVXYCDNWTXF744473KCBG3DAVQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.222079772 (593h48m36.093162807s) [noise=true , address=145.79.227.15:4876 , piece_id=G3OJRP6MW4LFWV4JNGSP7JSXZDUBDSAOHVJHGMTAYZGIBOUFJHUQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.310181082 (593h48m36.181264128s) [noise=true , address=145.79.227.46:4603 , piece_id=3753VEM45GQH6ZM3HQRPI4YNQKQGUX7HUXGDF2IAZH7SZLCXNPTQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.310200128 (593h48m36.181283163s) [seg_pos={0 0}, upload=520564]: Segment reached optimal threshold of 13 pieces. Dec 30 09:17:36.310360454 (593h48m36.181443489s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 30 09:17:36.310366275 (593h48m36.18144931s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310474010 (593h48m36.181557045s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 30 09:17:36.310482396 (593h48m36.181565431s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310557469 (593h48m36.181640504s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection Dec 30 09:17:36.310563931 (593h48m36.181646977s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310577156 (593h48m36.181660201s) [seg_pos={0 0}, upload=520564]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:17:36.310639726 (593h48m36.181722761s) [upload=520564]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:17:36.310643683 (593h48m36.181726718s) [upload=520564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:17:36.607104834 (593h48m36.478187869s) [upload=520564]: Done uploading segments... Dec 30 09:17:36.607124372 (593h48m36.478207407s) [upload=520564]: Upload finished. err: <nil> Duration: 6.470136762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.....................X..........................X....] duration: 43.335039589s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 43.33505624s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [....X....X.....X.........X...X.X....X.X...X........X.....X..] duration: 3.489414822s trace: https://jaeger.dev.storj.io/trace/731d29a169fda304 FAIL (2025-12-30 09:18:17 UTC) orbiter warning: context deadline exceeded Duration: 3.489480066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X...X] duration: 30.00567869s trace: https://jaeger.dev.storj.io/trace/11890a965976933 FAIL (2025-12-30 09:18:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A) context deadline exceeded; piecestore: (Node ID: 1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, Piece ID: VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA) context deadline exceeded; piecestore: (Node ID: 17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, Piece ID: WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA) context deadline exceeded ======LOGS====== 7:32.333269072 (593h48m32.204352117s) [upload=520564]: Starting upload Dec 30 09:17:32.333352361 (593h48m32.204435396s) [upload=520564]: Done starting upload Dec 30 09:17:32.333446972 (593h48m32.204530007s) [upload=520564]: Uploading segments... Dec 30 09:17:32.333686959 (593h48m32.204770004s) [upload=520564]: Got next segment. Inline: false Dec 30 09:17:32.333715523 (593h48m32.204798568s) [upload=520564]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:17:32.333722256 (593h48m32.204805301s) [upload=520564]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:17:32.581711087 (593h48m32.452794132s) [seg_pos={0 0}, upload=520564]: Begin upload segment... Dec 30 09:17:32.581787193 (593h48m32.452870238s) [seg_pos={0 0}, upload=520564]: Done begin upload segment. Dec 30 09:17:32.581816698 (593h48m32.452899743s) [noise=true , address=145.79.227.15:4876 , piece_id=G3OJRP6MW4LFWV4JNGSP7JSXZDUBDSAOHVJHGMTAYZGIBOUFJHUQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.581966834 (593h48m32.453049869s) [noise=true , address=145.79.227.36:4518 , piece_id=WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.582369851 (593h48m32.453452896s) [noise=true , address=145.79.227.50:4374 , piece_id=KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.583188921 (593h48m32.454271966s) [noise=true , address=145.79.227.31:4799 , piece_id=GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.583605244 (593h48m32.454688289s) [noise=true , address=145.79.227.32:4834 , piece_id=LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584119574 (593h48m32.455202609s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584506260 (593h48m32.455589295s) [noise=true , address=45.91.116.9:4262 , piece_id=4VSX2SZQD62B3T5KSQKQOE52DNYBC27S7FLH6EHZ6R2MPMW3A74Q, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.584880643 (593h48m32.455963678s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.585259754 (593h48m32.456342789s) [noise=true , address=145.79.227.14:4839 , piece_id=C55BUV5NDF3CEWZ3RNHAIDYXLMIVXYCDNWTXF744473KCBG3DAVQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.586159818 (593h48m32.457242853s) [noise=true , address=145.79.227.45:4229 , piece_id=SR3I6K7YMVIZXRZOOUE7TSQGZYHGSNF2EWHNUHGSHZV4HQPUOZHQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.588541834 (593h48m32.45962488s) [noise=true , address=145.79.227.11:4729 , piece_id=VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.591956266 (593h48m32.463039311s) [noise=true , address=77.90.31.18:4808 , piece_id=MU5OMFVVMLWQEYGF3YZFKFIFKZMKHCS62ZKQ3YMGWLT57DSOOR2A, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.593366611 (593h48m32.464449656s) [noise=true , address=145.79.227.9:4365 , piece_id=FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.594992598 (593h48m32.466075643s) [noise=true , address=77.90.31.17:4773 , piece_id=TYR46DEEB6MRGHVCJG4RGUMJGGNSPKGBKZADYO4OIP35JE76ZHVQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.596337158 (593h48m32.467420203s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:32.597969637 (593h48m32.469052682s) [noise=true , address=145.79.227.46:4603 , piece_id=3753VEM45GQH6ZM3HQRPI4YNQKQGUX7HUXGDF2IAZH7SZLCXNPTQ, seg_pos={0 0}, upload=520564]: Uploading piece... Dec 30 09:17:34.496610974 (593h48m34.367694009s) [upload=520564]: Next returned nil segment Dec 30 09:17:34.496641451 (593h48m34.367724486s) [upload=520564]: Waiting for error group managing segments... Dec 30 09:17:34.820502297 (593h48m34.691585332s) [noise=true , address=145.79.227.31:4799 , piece_id=GJSVAWO2XLKT7ITSRU67C4JD6JYK3AT7DTQG4LP4NIHH6ZVXXK5A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.827819970 (593h48m34.698903016s) [noise=true , address=145.79.227.32:4834 , piece_id=LNNVMHFQMXXTGZXQMNPZKAPSBDBYHZJPPUDLK2OT5QWHGPAGGN7A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.828989467 (593h48m34.700072502s) [noise=true , address=77.90.31.18:4808 , piece_id=MU5OMFVVMLWQEYGF3YZFKFIFKZMKHCS62ZKQ3YMGWLT57DSOOR2A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.849552623 (593h48m34.720635668s) [noise=true , address=145.79.227.11:4729 , piece_id=VRKEGQLQHCPSJ7R654MEJ5YKEDGI3DJNTWUEHE43BCDK6MQ2V34A, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.870010248 (593h48m34.741093293s) [noise=true , address=145.79.227.50:4374 , piece_id=KTJV2GDP57OIXOI7AGTCMHOE4LMDSQZNDZYWLVZZ5YJSBRQVOJTA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.887563172 (593h48m34.758646207s) [noise=true , address=45.91.116.9:4262 , piece_id=4VSX2SZQD62B3T5KSQKQOE52DNYBC27S7FLH6EHZ6R2MPMW3A74Q, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:34.981545660 (593h48m34.852628695s) [noise=true , address=77.90.31.17:4773 , piece_id=TYR46DEEB6MRGHVCJG4RGUMJGGNSPKGBKZADYO4OIP35JE76ZHVQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.484542950 (593h48m35.355625985s) [noise=true , address=145.79.227.45:4229 , piece_id=SR3I6K7YMVIZXRZOOUE7TSQGZYHGSNF2EWHNUHGSHZV4HQPUOZHQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.507515103 (593h48m35.378598148s) [noise=true , address=145.79.227.9:4365 , piece_id=FATIUCWCWW7G6SUBLHN5XWAIYCTGIIHJISBLQDO5MMDSAMP3OZFA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:35.573332239 (593h48m35.444415274s) [noise=true , address=145.79.227.36:4518 , piece_id=WW4EY3PGMVUE36U55IBWQNZANB7RQHH6FRWABDFPQGVQONU7AIBA, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.083563394 (593h48m35.954646429s) [noise=true , address=145.79.227.14:4839 , piece_id=C55BUV5NDF3CEWZ3RNHAIDYXLMIVXYCDNWTXF744473KCBG3DAVQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.222079772 (593h48m36.093162807s) [noise=true , address=145.79.227.15:4876 , piece_id=G3OJRP6MW4LFWV4JNGSP7JSXZDUBDSAOHVJHGMTAYZGIBOUFJHUQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.310181082 (593h48m36.181264128s) [noise=true , address=145.79.227.46:4603 , piece_id=3753VEM45GQH6ZM3HQRPI4YNQKQGUX7HUXGDF2IAZH7SZLCXNPTQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: <nil> Dec 30 09:17:36.310200128 (593h48m36.181283163s) [seg_pos={0 0}, upload=520564]: Segment reached optimal threshold of 13 pieces. Dec 30 09:17:36.310360454 (593h48m36.181443489s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 30 09:17:36.310366275 (593h48m36.18144931s) [noise=true , address=145.79.227.42:4121 , piece_id=C5R3XQPHTYQD3SZJSED7XBWDYPARDXFKAIP7VYS4FDS5TYIIJJCQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310474010 (593h48m36.181557045s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 30 09:17:36.310482396 (593h48m36.181565431s) [noise=true , address=145.79.227.37:4565 , piece_id=SOW7KBLLTSVRYFQ2VBQA7H6XUN6ATIZ4X624KH7JG3H5R2RBL7CQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310557469 (593h48m36.181640504s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56898->145.79.227.39:4013: use of closed network connection Dec 30 09:17:36.310563931 (593h48m36.181646977s) [noise=true , address=145.79.227.39:4013 , piece_id=SJJEHRMCEFFTHN24QBM45HZC5D7JKQIJHBKAXY62VQT5YEJDO6KQ, seg_pos={0 0}, upload=520564]: Long tail cancellation due to optimal threshold Dec 30 09:17:36.310577156 (593h48m36.181660201s) [seg_pos={0 0}, upload=520564]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:17:36.310639726 (593h48m36.181722761s) [upload=520564]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:17:36.310643683 (593h48m36.181726718s) [upload=520564]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:17:36.607104834 (593h48m36.478187869s) [upload=520564]: Done uploading segments... Dec 30 09:17:36.607124372 (593h48m36.478207407s) [upload=520564]: Upload finished. err: <nil> Duration: 30.005700781s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.....X....X.....X.........X...X.X....X.X...X........X.....X.] duration: 3.719595806s trace: https://jaeger.dev.storj.io/trace/731d29a169fda304 FAIL (2025-12-30 09:17:48 UTC) orbiter warning: context deadline exceeded Duration: 3.719623048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.....................X..........................X...] duration: 48.288315872s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:17:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 48.288338526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X...] duration: 5.867540085s trace: https://jaeger.dev.storj.io/trace/1acb904f4327d6eb FAIL (2025-12-30 09:16:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ) context deadline exceeded ======LOGS====== 6m22.461911784s) [upload=520534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:15:22.838951015 (593h46m22.71003405s) [seg_pos={0 0}, upload=520534]: Begin upload segment... Dec 30 09:15:22.839005920 (593h46m22.710088965s) [seg_pos={0 0}, upload=520534]: Done begin upload segment. Dec 30 09:15:22.839031097 (593h46m22.710114132s) [noise=true , address=145.79.227.38:4978 , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839044543 (593h46m22.710127578s) [noise=true , address=145.79.227.29:4943 , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839106782 (593h46m22.710189827s) [noise=true , address=145.79.227.8:4330 , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839547971 (593h46m22.710631006s) [noise=true , address=145.79.227.21:4473 , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840120452 (593h46m22.711203497s) [noise=true , address=145.79.227.41:4085 , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840485116 (593h46m22.711568161s) [noise=true , address=145.79.227.3:4149 , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840851314 (593h46m22.711934359s) [noise=true , address=145.79.227.16:4911 , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.841336968 (593h46m22.712420014s) [noise=true , address=77.90.31.19:4842 , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.841778589 (593h46m22.712861634s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842371728 (593h46m22.713454773s) [noise=true , address=45.91.116.11:4118 , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842656211 (593h46m22.713739246s) [noise=true , address=45.91.116.5:4299 , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842776199 (593h46m22.713859244s) [noise=true , address=145.79.227.35:4482 , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.843303513 (593h46m22.714386548s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.843975492 (593h46m22.715058537s) [noise=true , address=145.79.227.25:4619 , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.844415250 (593h46m22.715498295s) [noise=true , address=145.79.227.45:4230 , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.844815763 (593h46m22.715898798s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:24.547256030 (593h46m24.418339065s) [upload=520534]: Next returned nil segment Dec 30 09:15:24.547272341 (593h46m24.418355386s) [upload=520534]: Waiting for error group managing segments... Dec 30 09:15:24.826215676 (593h46m24.697298721s) [noise=true , address=145.79.227.41:4085 , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.828294303 (593h46m24.699377349s) [noise=true , address=145.79.227.29:4943 , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.829024574 (593h46m24.700107619s) [noise=true , address=145.79.227.21:4473 , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.830783313 (593h46m24.701866359s) [noise=true , address=145.79.227.3:4149 , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.831242437 (593h46m24.702325473s) [noise=true , address=145.79.227.35:4482 , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.844413818 (593h46m24.715496853s) [noise=true , address=145.79.227.8:4330 , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.846797357 (593h46m24.717880392s) [noise=true , address=145.79.227.38:4978 , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.847348827 (593h46m24.718431872s) [noise=true , address=145.79.227.25:4619 , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.851805403 (593h46m24.722888438s) [noise=true , address=145.79.227.16:4911 , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.921016151 (593h46m24.792099186s) [noise=true , address=145.79.227.45:4230 , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.037264077 (593h46m24.908347112s) [noise=true , address=77.90.31.19:4842 , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.231078461 (593h46m25.102161496s) [noise=true , address=45.91.116.5:4299 , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.284792041 (593h46m25.155875086s) [noise=true , address=45.91.116.11:4118 , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.284817389 (593h46m25.155913139s) [seg_pos={0 0}, upload=520534]: Segment reached optimal threshold of 13 pieces. Dec 30 09:15:25.285025375 (593h46m25.15610841s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection Dec 30 09:15:25.285033050 (593h46m25.156116085s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285113102 (593h46m25.156196157s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection Dec 30 09:15:25.285126828 (593h46m25.156209873s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285134242 (593h46m25.156217277s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection Dec 30 09:15:25.285139693 (593h46m25.156222728s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285160132 (593h46m25.156243167s) [seg_pos={0 0}, upload=520534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:15:25.285228702 (593h46m25.156311737s) [upload=520534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:15:25.285233631 (593h46m25.156316687s) [upload=520534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:15:25.582418231 (593h46m25.453501276s) [upload=520534]: Done uploading segments... Dec 30 09:15:25.582450762 (593h46m25.453533797s) [upload=520534]: Upload finished. err: <nil> Duration:* 5.86755898s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [......X....X.....X.........X...X.X....X.X...X........X.....X] duration: 5.000815056s trace: https://jaeger.dev.storj.io/trace/731d29a169fda304 FAIL (2025-12-30 09:16:39 UTC) orbiter warning: context deadline exceeded Duration: 5.000836197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X..] duration: 5.969337276s trace: https://jaeger.dev.storj.io/trace/1acb904f4327d6eb FAIL (2025-12-30 09:16:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ) context deadline exceeded ======LOGS====== 6m22.461911784s) [upload=520534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:15:22.838951015 (593h46m22.71003405s) [seg_pos={0 0}, upload=520534]: Begin upload segment... Dec 30 09:15:22.839005920 (593h46m22.710088965s) [seg_pos={0 0}, upload=520534]: Done begin upload segment. Dec 30 09:15:22.839031097 (593h46m22.710114132s) [noise=true , address=145.79.227.38:4978 , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839044543 (593h46m22.710127578s) [noise=true , address=145.79.227.29:4943 , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839106782 (593h46m22.710189827s) [noise=true , address=145.79.227.8:4330 , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839547971 (593h46m22.710631006s) [noise=true , address=145.79.227.21:4473 , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840120452 (593h46m22.711203497s) [noise=true , address=145.79.227.41:4085 , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840485116 (593h46m22.711568161s) [noise=true , address=145.79.227.3:4149 , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840851314 (593h46m22.711934359s) [noise=true , address=145.79.227.16:4911 , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.841336968 (593h46m22.712420014s) [noise=true , address=77.90.31.19:4842 , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.841778589 (593h46m22.712861634s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842371728 (593h46m22.713454773s) [noise=true , address=45.91.116.11:4118 , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842656211 (593h46m22.713739246s) [noise=true , address=45.91.116.5:4299 , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842776199 (593h46m22.713859244s) [noise=true , address=145.79.227.35:4482 , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.843303513 (593h46m22.714386548s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.843975492 (593h46m22.715058537s) [noise=true , address=145.79.227.25:4619 , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.844415250 (593h46m22.715498295s) [noise=true , address=145.79.227.45:4230 , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.844815763 (593h46m22.715898798s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:24.547256030 (593h46m24.418339065s) [upload=520534]: Next returned nil segment Dec 30 09:15:24.547272341 (593h46m24.418355386s) [upload=520534]: Waiting for error group managing segments... Dec 30 09:15:24.826215676 (593h46m24.697298721s) [noise=true , address=145.79.227.41:4085 , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.828294303 (593h46m24.699377349s) [noise=true , address=145.79.227.29:4943 , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.829024574 (593h46m24.700107619s) [noise=true , address=145.79.227.21:4473 , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.830783313 (593h46m24.701866359s) [noise=true , address=145.79.227.3:4149 , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.831242437 (593h46m24.702325473s) [noise=true , address=145.79.227.35:4482 , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.844413818 (593h46m24.715496853s) [noise=true , address=145.79.227.8:4330 , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.846797357 (593h46m24.717880392s) [noise=true , address=145.79.227.38:4978 , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.847348827 (593h46m24.718431872s) [noise=true , address=145.79.227.25:4619 , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.851805403 (593h46m24.722888438s) [noise=true , address=145.79.227.16:4911 , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.921016151 (593h46m24.792099186s) [noise=true , address=145.79.227.45:4230 , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.037264077 (593h46m24.908347112s) [noise=true , address=77.90.31.19:4842 , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.231078461 (593h46m25.102161496s) [noise=true , address=45.91.116.5:4299 , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.284792041 (593h46m25.155875086s) [noise=true , address=45.91.116.11:4118 , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.284817389 (593h46m25.155913139s) [seg_pos={0 0}, upload=520534]: Segment reached optimal threshold of 13 pieces. Dec 30 09:15:25.285025375 (593h46m25.15610841s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection Dec 30 09:15:25.285033050 (593h46m25.156116085s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285113102 (593h46m25.156196157s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection Dec 30 09:15:25.285126828 (593h46m25.156209873s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285134242 (593h46m25.156217277s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection Dec 30 09:15:25.285139693 (593h46m25.156222728s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285160132 (593h46m25.156243167s) [seg_pos={0 0}, upload=520534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:15:25.285228702 (593h46m25.156311737s) [upload=520534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:15:25.285233631 (593h46m25.156316687s) [upload=520534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:15:25.582418231 (593h46m25.453501276s) [upload=520534]: Done uploading segments... Dec 30 09:15:25.582450762 (593h46m25.453533797s) [upload=520534]: Upload finished. err: <nil> Duration:* 5.969364558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X.] duration: 6.119780896s trace: https://jaeger.dev.storj.io/trace/1acb904f4327d6eb FAIL (2025-12-30 09:15:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ) context deadline exceeded ======LOGS====== 6m22.461911784s) [upload=520534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:15:22.838951015 (593h46m22.71003405s) [seg_pos={0 0}, upload=520534]: Begin upload segment... Dec 30 09:15:22.839005920 (593h46m22.710088965s) [seg_pos={0 0}, upload=520534]: Done begin upload segment. Dec 30 09:15:22.839031097 (593h46m22.710114132s) [noise=true , address=145.79.227.38:4978 , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839044543 (593h46m22.710127578s) [noise=true , address=145.79.227.29:4943 , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839106782 (593h46m22.710189827s) [noise=true , address=145.79.227.8:4330 , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839547971 (593h46m22.710631006s) [noise=true , address=145.79.227.21:4473 , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840120452 (593h46m22.711203497s) [noise=true , address=145.79.227.41:4085 , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840485116 (593h46m22.711568161s) [noise=true , address=145.79.227.3:4149 , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840851314 (593h46m22.711934359s) [noise=true , address=145.79.227.16:4911 , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.841336968 (593h46m22.712420014s) [noise=true , address=77.90.31.19:4842 , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.841778589 (593h46m22.712861634s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842371728 (593h46m22.713454773s) [noise=true , address=45.91.116.11:4118 , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842656211 (593h46m22.713739246s) [noise=true , address=45.91.116.5:4299 , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842776199 (593h46m22.713859244s) [noise=true , address=145.79.227.35:4482 , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.843303513 (593h46m22.714386548s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.843975492 (593h46m22.715058537s) [noise=true , address=145.79.227.25:4619 , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.844415250 (593h46m22.715498295s) [noise=true , address=145.79.227.45:4230 , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.844815763 (593h46m22.715898798s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:24.547256030 (593h46m24.418339065s) [upload=520534]: Next returned nil segment Dec 30 09:15:24.547272341 (593h46m24.418355386s) [upload=520534]: Waiting for error group managing segments... Dec 30 09:15:24.826215676 (593h46m24.697298721s) [noise=true , address=145.79.227.41:4085 , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.828294303 (593h46m24.699377349s) [noise=true , address=145.79.227.29:4943 , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.829024574 (593h46m24.700107619s) [noise=true , address=145.79.227.21:4473 , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.830783313 (593h46m24.701866359s) [noise=true , address=145.79.227.3:4149 , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.831242437 (593h46m24.702325473s) [noise=true , address=145.79.227.35:4482 , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.844413818 (593h46m24.715496853s) [noise=true , address=145.79.227.8:4330 , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.846797357 (593h46m24.717880392s) [noise=true , address=145.79.227.38:4978 , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.847348827 (593h46m24.718431872s) [noise=true , address=145.79.227.25:4619 , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.851805403 (593h46m24.722888438s) [noise=true , address=145.79.227.16:4911 , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.921016151 (593h46m24.792099186s) [noise=true , address=145.79.227.45:4230 , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.037264077 (593h46m24.908347112s) [noise=true , address=77.90.31.19:4842 , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.231078461 (593h46m25.102161496s) [noise=true , address=45.91.116.5:4299 , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.284792041 (593h46m25.155875086s) [noise=true , address=45.91.116.11:4118 , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.284817389 (593h46m25.155913139s) [seg_pos={0 0}, upload=520534]: Segment reached optimal threshold of 13 pieces. Dec 30 09:15:25.285025375 (593h46m25.15610841s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection Dec 30 09:15:25.285033050 (593h46m25.156116085s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285113102 (593h46m25.156196157s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection Dec 30 09:15:25.285126828 (593h46m25.156209873s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285134242 (593h46m25.156217277s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection Dec 30 09:15:25.285139693 (593h46m25.156222728s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285160132 (593h46m25.156243167s) [seg_pos={0 0}, upload=520534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:15:25.285228702 (593h46m25.156311737s) [upload=520534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:15:25.285233631 (593h46m25.156316687s) [upload=520534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:15:25.582418231 (593h46m25.453501276s) [upload=520534]: Done uploading segments... Dec 30 09:15:25.582450762 (593h46m25.453533797s) [upload=520534]: Upload finished. err: <nil> Duration:* 6.119795533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.....................X..........................X..] duration: 15.264178044s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:15:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 15.264195738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......X] duration: 30.00218075s trace: https://jaeger.dev.storj.io/trace/1acb904f4327d6eb FAIL (2025-12-30 09:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ) context deadline exceeded ======LOGS====== 6m22.461911784s) [upload=520534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:15:22.838951015 (593h46m22.71003405s) [seg_pos={0 0}, upload=520534]: Begin upload segment... Dec 30 09:15:22.839005920 (593h46m22.710088965s) [seg_pos={0 0}, upload=520534]: Done begin upload segment. Dec 30 09:15:22.839031097 (593h46m22.710114132s) [noise=true , address=145.79.227.38:4978 , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839044543 (593h46m22.710127578s) [noise=true , address=145.79.227.29:4943 , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839106782 (593h46m22.710189827s) [noise=true , address=145.79.227.8:4330 , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.839547971 (593h46m22.710631006s) [noise=true , address=145.79.227.21:4473 , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840120452 (593h46m22.711203497s) [noise=true , address=145.79.227.41:4085 , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840485116 (593h46m22.711568161s) [noise=true , address=145.79.227.3:4149 , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.840851314 (593h46m22.711934359s) [noise=true , address=145.79.227.16:4911 , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.841336968 (593h46m22.712420014s) [noise=true , address=77.90.31.19:4842 , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.841778589 (593h46m22.712861634s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842371728 (593h46m22.713454773s) [noise=true , address=45.91.116.11:4118 , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842656211 (593h46m22.713739246s) [noise=true , address=45.91.116.5:4299 , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.842776199 (593h46m22.713859244s) [noise=true , address=145.79.227.35:4482 , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.843303513 (593h46m22.714386548s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.843975492 (593h46m22.715058537s) [noise=true , address=145.79.227.25:4619 , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.844415250 (593h46m22.715498295s) [noise=true , address=145.79.227.45:4230 , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:22.844815763 (593h46m22.715898798s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Uploading piece... Dec 30 09:15:24.547256030 (593h46m24.418339065s) [upload=520534]: Next returned nil segment Dec 30 09:15:24.547272341 (593h46m24.418355386s) [upload=520534]: Waiting for error group managing segments... Dec 30 09:15:24.826215676 (593h46m24.697298721s) [noise=true , address=145.79.227.41:4085 , piece_id=SNEX4LZMPARHKIL6LWMLOFRYE2RDOTR6ROWNM3W6IFCNPQFG3JSQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.828294303 (593h46m24.699377349s) [noise=true , address=145.79.227.29:4943 , piece_id=33S2QGMW7F2TW3TIBA3XG7KRLD2FAP4O6ASJIBT75TK7BIHKO4TQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.829024574 (593h46m24.700107619s) [noise=true , address=145.79.227.21:4473 , piece_id=55J3SBHLQ3X5MKERQWT4F6FT45LYOSK45SNM2BKNVQJDKAYWU7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.830783313 (593h46m24.701866359s) [noise=true , address=145.79.227.3:4149 , piece_id=5ACEQGCPZGML4ZTZJ6ENKFWC7GMCZFMBPRZDWMIOYZTDXM2GIWVQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.831242437 (593h46m24.702325473s) [noise=true , address=145.79.227.35:4482 , piece_id=RKJMZCKDRMZQ5YCYNMVAUX3RPD2OXG5RFMKOT7AXRMS466CLG7PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.844413818 (593h46m24.715496853s) [noise=true , address=145.79.227.8:4330 , piece_id=LNITVNNEX7PGXH7OLHU4VKPXQKLZYTMSB2FN3GXLMWFSDYY5B3BQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.846797357 (593h46m24.717880392s) [noise=true , address=145.79.227.38:4978 , piece_id=JUUCWZS57A52OTLJDM2GHLGRO22X2ZJJG24QKTJ5S5AE7EMQ4IFA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.847348827 (593h46m24.718431872s) [noise=true , address=145.79.227.25:4619 , piece_id=TAIRTUPOOPUXYSJIZ7AUS2SW4T3E6OJPHXUUV5UV3BT3MYLHBRTA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.851805403 (593h46m24.722888438s) [noise=true , address=145.79.227.16:4911 , piece_id=C6JATSA6NFPEYIJ5MMKYOKYYWCQWYW2QSX3NA4OW2SSUINGJQUKQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:24.921016151 (593h46m24.792099186s) [noise=true , address=145.79.227.45:4230 , piece_id=H5Z6EXEIS7VCAOI2577XBPKYT35ZHU7BMJBHNS2B4MV5PSCL46PQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.037264077 (593h46m24.908347112s) [noise=true , address=77.90.31.19:4842 , piece_id=B2ZNLI3WHRSKCUX2OFBVX6UTMATKPAH5F4TYVGVYI64FRYSL7C4A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.231078461 (593h46m25.102161496s) [noise=true , address=45.91.116.5:4299 , piece_id=2SGO63LND473C6LA5RVXBHGDABHUHXYWUQQRN5MSJJ7YVNCCB5CQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.284792041 (593h46m25.155875086s) [noise=true , address=45.91.116.11:4118 , piece_id=OLNBNSIZGXYY7XA4RTCRV2M5QTXKFXUIKY72OGHWPJTMFCYEHWFQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: <nil> Dec 30 09:15:25.284817389 (593h46m25.155913139s) [seg_pos={0 0}, upload=520534]: Segment reached optimal threshold of 13 pieces. Dec 30 09:15:25.285025375 (593h46m25.15610841s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43798->45.91.116.14:4192: use of closed network connection Dec 30 09:15:25.285033050 (593h46m25.156116085s) [noise=true , address=45.91.116.14:4192 , piece_id=DQOK44IEFPWPV52Y72C2IYQ37TALYDJ2SXRJPK5UKFOQLTW56MPA, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285113102 (593h46m25.156196157s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50674->145.79.227.51:4410: use of closed network connection Dec 30 09:15:25.285126828 (593h46m25.156209873s) [noise=true , address=145.79.227.51:4410 , piece_id=2MS6ZYECPN5SK37O4SN63LRAX63VSU6O6NIUMTFCWNOG36EHV2EQ, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285134242 (593h46m25.156217277s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50704->145.79.227.42:4122: use of closed network connection Dec 30 09:15:25.285139693 (593h46m25.156222728s) [noise=true , address=145.79.227.42:4122 , piece_id=BU7T4VNOUO3O65BCNF5BGIO3ZPGGMBHFWRDFRVTNUL3VZIAY2U6A, seg_pos={0 0}, upload=520534]: Long tail cancellation due to optimal threshold Dec 30 09:15:25.285160132 (593h46m25.156243167s) [seg_pos={0 0}, upload=520534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:15:25.285228702 (593h46m25.156311737s) [upload=520534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:15:25.285233631 (593h46m25.156316687s) [upload=520534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:15:25.582418231 (593h46m25.453501276s) [upload=520534]: Done uploading segments... Dec 30 09:15:25.582450762 (593h46m25.453533797s) [upload=520534]: Upload finished. err: <nil> Duration:* 30.002206951s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......X....X.....X.........X...X.X....X.X...X........X.....] duration: 5.000705993s trace: https://jaeger.dev.storj.io/trace/30e1221c2da57697 FAIL (2025-12-30 09:15:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000731912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [........X....X.....X.........X...X.X....X.X...X........X....] duration: 5.000957408s trace: https://jaeger.dev.storj.io/trace/30e1221c2da57697 FAIL (2025-12-30 09:15:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000983898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.....................X..........................X.] duration: 21.518248392s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:14:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 21.51826399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX......] duration: 5.716469634s trace: https://jaeger.dev.storj.io/trace/62ea86697d2d9b63 FAIL (2025-12-30 09:14:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments... Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment... Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment. Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments... Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces. Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments... Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil> Duration: 5.716497617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.....................X..........................X] duration: 59.244007161s trace: https://jaeger.dev.storj.io/trace/18468de8cb6443cc FAIL (2025-12-30 09:14:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== oise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797757101 (593h44m35.668840147s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.797860649 (593h44m35.668943684s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798202941 (593h44m35.669285986s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798925637 (593h44m35.670008682s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.798986884 (593h44m35.670069929s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800365068 (593h44m35.671448113s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.800566742 (593h44m35.671649797s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.801577667 (593h44m35.672660712s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.802176186 (593h44m35.673259221s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803109253 (593h44m35.674192298s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.803705689 (593h44m35.674788734s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804199369 (593h44m35.675282414s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.804782018 (593h44m35.675865063s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.805338889 (593h44m35.676421924s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:35.806099477 (593h44m35.677182522s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Uploading piece... Dec 30 09:13:37.193520576 (593h44m37.064603611s) [upload=520508]: Next returned nil segment Dec 30 09:13:37.193538250 (593h44m37.064621286s) [upload=520508]: Waiting for error group managing segments... Dec 30 09:13:37.645471907 (593h44m37.516554942s) [noise=true , address=145.79.227.43:4174 , piece_id=Z7LBHTWYLKPJ5D33A3VP5KF74AQD5UTOAWVS34Y5I7TTRNXF4O6Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.687503351 (593h44m37.558586396s) [noise=true , address=145.79.227.21:4495 , piece_id=XQNSN2JJ2BSVGZ5KL3GCG2CTYKAK7LFZQDKKLQTHFRTSHI6FJ4SA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.692513832 (593h44m37.563596867s) [noise=true , address=145.79.227.11:4758 , piece_id=VPO5LVT2V2DIV3TSKLKC4IWFDB5NMBRWR4O7YU7P3ZRCAIEBUZCA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.697589678 (593h44m37.568672713s) [noise=true , address=145.79.227.19:4106 , piece_id=SGIFSSBSOI7JZF6YMCMY4CFXYFA56LDAV7Y54NJHKAF6P5FYZHLQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.704278705 (593h44m37.57536175s) [noise=true , address=145.79.227.49:4347 , piece_id=GM3PHNBZ44K7KRBEBHU3F5F7BDXZA4GYCOHD2BTCLDBH3RY4KD6A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.749865058 (593h44m37.620948103s) [noise=true , address=145.79.227.29:4967 , piece_id=6J76RHC2W5XSJIRXJF72RLWVNYE3HHXRZJS2FFAQDFHJTDVKITUQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.759243937 (593h44m37.630326982s) [noise=true , address=145.79.227.35:4516 , piece_id=BYRER7S6ROCRK5O6DQWSRMUGQVNESCF2OTANF54BHPNDASRH4XLA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.798175980 (593h44m37.669259025s) [noise=true , address=145.79.227.40:4060 , piece_id=3RXJLSLBYZSG4DSTN6XQRFKFCCZDRVNIKYRESQKL5X7VTBJ6J2LA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.890268539 (593h44m37.761351575s) [noise=true , address=77.90.31.19:4862 , piece_id=FGU6A4Q3CFW5RG6FHWAVV63OGW5YHT53NTGIQ6BSA3JQAPBY5RWA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.952799138 (593h44m37.823882183s) [noise=true , address=77.90.31.17:4779 , piece_id=JMTH7R7F7BMMZHZJMC7JLWIW2UDPOTX32ETZNYBGCPNPSQ24YOYQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:37.963189332 (593h44m37.834272368s) [noise=true , address=77.90.31.15:4714 , piece_id=QHQUOFKLG6UD5IAMOUWHHYHWACYMJOJWY4HJG5RQGTVW6TXQA7HQ, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.436254788 (593h44m38.307337823s) [noise=true , address=193.149.164.143:4700 , piece_id=S75DOXG3IIUHMS7XA3CFEFSXOAJFZUBRP3XQTLD6TCNQQYXI6M7A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.549026644 (593h44m38.420109679s) [noise=true , address=145.79.227.44:4203 , piece_id=EPIDBLV22RONSI7SLP4QAKYAL4DPQJ7ZQB7XYES4AGL4WCA2GQMA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640186527 (593h44m38.511269572s) [noise=true , address=145.79.227.30:4785 , piece_id=4BFXACOTE2VX4277YAO7A26Y7CLIYI4XPNITRLJJJLSY5QICNKSA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: <nil> Dec 30 09:13:38.640207968 (593h44m38.511291003s) [seg_pos={0 0}, upload=520508]: Segment reached optimal threshold of 14 pieces. Dec 30 09:13:38.640431473 (593h44m38.511514519s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45640->64.49.13.169:4710: use of closed network connection Dec 30 09:13:38.640438788 (593h44m38.511521823s) [noise=true , address=64.49.13.169:4710 , piece_id=MHOSB3NBMV7VMVBQS2V4BV2ZY5OK2PUNMBJB3NFICALO2KR6YZAA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.640811708 (593h44m38.511894753s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RRHuCjhncTpv9rqiUxiCdvzvzVU3RiYHm2NZY3BydASKZVLua): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.640818591 (593h44m38.511901627s) [noise=true , address=45.91.116.14:4215 , piece_id=6WIRCINNKLZDS3SFYZBJ7K3H4CMMO4YYCYCWFZRK4APPPTXQE23A, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641026507 (593h44m38.512109542s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641032569 (593h44m38.512115614s) [noise=true , address=145.79.227.33:4886 , piece_id=JKR5KL5BFW5RMCR2XSRRVAYHGOPJ2EFWDXLSKTLXT23AEKGJIHHA, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641210668 (593h44m38.512293713s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 30 09:13:38.641217130 (593h44m38.512300175s) [noise=true , address=77.90.31.13:4649 , piece_id=U6GILUWVKHBGQJRMCLR42PCCD5JBVVMKFHXZ7URM63W2LW72JG2Q, seg_pos={0 0}, upload=520508]: Long tail cancellation due to optimal threshold Dec 30 09:13:38.641234433 (593h44m38.512317478s) [seg_pos={0 0}, upload=520508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 09:13:38.641332840 (593h44m38.512415875s) [upload=520508]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:13:38.641337378 (593h44m38.512420423s) [upload=520508]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:13:38.925912658 (593h44m38.796995723s) [upload=520508]: Done uploading segments... Dec 30 09:13:38.925949849 (593h44m38.797032894s) [upload=520508]: Upload finished. err: <nil> Duration: 59.24403311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........X....X.....X.........X...X.X....X.X...X........X...] duration: 3.66347834s trace: https://jaeger.dev.storj.io/trace/30e1221c2da57697 FAIL (2025-12-30 09:14:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.663498286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX.....] duration: 6.182399584s trace: https://jaeger.dev.storj.io/trace/62ea86697d2d9b63 FAIL (2025-12-30 09:13:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments... Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment... Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment. Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments... Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces. Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments... Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil> Duration: 6.182413851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX....] duration: 6.651237185s trace: https://jaeger.dev.storj.io/trace/62ea86697d2d9b63 FAIL (2025-12-30 09:13:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments... Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment... Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment. Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments... Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces. Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments... Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil> Duration: 6.651264749s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........X....X.....X.........X...X.X....X.X...X........X..] duration: 3.546461399s trace: https://jaeger.dev.storj.io/trace/30e1221c2da57697 FAIL (2025-12-30 09:13:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546489333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.....................X..........................] duration: 29.469710686s trace: https://jaeger.dev.storj.io/trace/e14e3f77cfefb30 FAIL (2025-12-30 09:13:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments... Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956 , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594 , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713 , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces. Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments... Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil> Duration: 29.46973963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX...] duration: 5.825308973s trace: https://jaeger.dev.storj.io/trace/62ea86697d2d9b63 FAIL (2025-12-30 09:13:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments... Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment... Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment. Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments... Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces. Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments... Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil> Duration: 5.82533903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX..] duration: 6.144246651s trace: https://jaeger.dev.storj.io/trace/62ea86697d2d9b63 FAIL (2025-12-30 09:12:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments... Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment... Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment. Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments... Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces. Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments... Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil> Duration: 6.144274103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.....................X.........................] duration: 38.66283598s trace: https://jaeger.dev.storj.io/trace/e14e3f77cfefb30 FAIL (2025-12-30 09:12:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments... Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956 , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594 , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713 , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces. Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments... Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil> Duration: 38.662866828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........X....X.....X.........X...X.X....X.X...X........X.] duration: 3.552764963s trace: https://jaeger.dev.storj.io/trace/30e1221c2da57697 FAIL (2025-12-30 09:12:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552793148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX.] duration: 5.611117266s trace: https://jaeger.dev.storj.io/trace/62ea86697d2d9b63 FAIL (2025-12-30 09:11:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments... Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment... Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment. Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments... Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces. Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments... Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil> Duration: 5.611131464s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [............X....X.....X.........X...X.X....X.X...X........X] duration: 5.001180518s trace: https://jaeger.dev.storj.io/trace/30e1221c2da57697 FAIL (2025-12-30 09:11:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001206657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.....................X........................] duration: 38.287689605s trace: https://jaeger.dev.storj.io/trace/e14e3f77cfefb30 FAIL (2025-12-30 09:11:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments... Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956 , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594 , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713 , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces. Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments... Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil> Duration: 38.287721146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..XX] duration: 30.003351355s trace: https://jaeger.dev.storj.io/trace/62ea86697d2d9b63 FAIL (2025-12-30 09:10:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 09:10:12.814668123 (593h41m12.685751168s) [upload=520457]: Starting upload Dec 30 09:10:12.814754999 (593h41m12.685838034s) [upload=520457]: Done starting upload Dec 30 09:10:12.814789886 (593h41m12.685872921s) [upload=520457]: Uploading segments... Dec 30 09:10:12.814982211 (593h41m12.686065256s) [upload=520457]: Got next segment. Inline: false Dec 30 09:10:12.815021907 (593h41m12.686104952s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:10:12.815027688 (593h41m12.686110733s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:10:13.060347999 (593h41m12.931431034s) [seg_pos={0 0}, upload=520457]: Begin upload segment... Dec 30 09:10:13.060402241 (593h41m12.931485286s) [seg_pos={0 0}, upload=520457]: Done begin upload segment. Dec 30 09:10:13.060435645 (593h41m12.93151869s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.060473166 (593h41m12.931556211s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061018024 (593h41m12.932101079s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061530520 (593h41m12.932613565s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.061964426 (593h41m12.933047471s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062433951 (593h41m12.933516986s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.062726908 (593h41m12.933809953s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063030316 (593h41m12.934113351s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.063403988 (593h41m12.934487033s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065410909 (593h41m12.936493944s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065436498 (593h41m12.936519543s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.065516039 (593h41m12.936599074s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.068232482 (593h41m12.939315517s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.070517783 (593h41m12.941600819s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.074091739 (593h41m12.945174784s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:13.075655647 (593h41m12.946738702s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Uploading piece... Dec 30 09:10:14.767368689 (593h41m14.638451734s) [upload=520457]: Next returned nil segment Dec 30 09:10:14.767390469 (593h41m14.638473504s) [upload=520457]: Waiting for error group managing segments... Dec 30 09:10:15.049098589 (593h41m14.920181634s) [noise=true , address=145.79.227.6:4257 , piece_id=DV4SYWFS6KLNMCL5ZGTWRNE6PSDS764GGJ7MSM2W2SXY365ARF2Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.054937517 (593h41m14.926020552s) [noise=true , address=145.79.227.45:4229 , piece_id=JZWD7BBFAAC5W7WTJRRPYZ3SKLVXDDBLLKFTS3CSI4P65GGMOYVA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.060731359 (593h41m14.931814404s) [noise=true , address=145.79.227.15:4876 , piece_id=U4N6H6Q5WCOFUXQG6SPTBNFWM65LUML5PIFO55S4RMIVCDHEYSXQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.069240191 (593h41m14.940323226s) [noise=true , address=145.79.227.32:4834 , piece_id=R552YAMSKP5JSGQOQ6WTOYV3JK7QD2AOABGICIHEVP4OLSV4HXYA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.071869268 (593h41m14.942952303s) [noise=true , address=145.79.227.4:4186 , piece_id=S6RR7AWPEUEIJIFT6ZL4LENQ4XCHTROWMVAY6OEU5Z5I654Q6BQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.072146786 (593h41m14.943229821s) [noise=true , address=145.79.227.51:4411 , piece_id=M4ZIOBGAKQKX54KPKQ3PIILF43RCDSBAJYBMQOPJID6XII6KGQ4Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.080132021 (593h41m14.951215056s) [noise=true , address=145.79.227.11:4729 , piece_id=6LKDAMVMMGZPVYDBLQ7ZWBHVL22OO2IQNSNDPTYPOZMRM5WHTY5Q, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.088523490 (593h41m14.959606535s) [noise=true , address=145.79.227.36:4518 , piece_id=DMW7DN6DJXAIXHPV7HDZDBI3Z7U76VF6QPQWDNO4A3QJYRLTZRUQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.334082706 (593h41m15.205165741s) [noise=true , address=77.90.31.17:4770 , piece_id=N7DP7N6VLQV4AR5TJOJQTGBX7H7ZCH7AWUBLXXRUMPQJMQGUP2RQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.337976009 (593h41m15.209059044s) [noise=true , address=77.90.31.16:4736 , piece_id=3UEAYGJDMV4425RNENQYKLLZOW6DHZ7BLCJRQVHPCFJRHCVOAXOA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.392914320 (593h41m15.263997356s) [noise=true , address=45.91.116.6:4974 , piece_id=SRECJQATG7Z7CAP5ONUTSR4XDZNDDHFGEFVO3GG636DFRUKYPBJA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.436704074 (593h41m15.307787119s) [noise=true , address=45.91.116.9:4261 , piece_id=L4HJSL6TMIMNWF72XC2SSSEDI4AADC7HKSH3PYKFWQY6ORCRTG6A, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496262382 (593h41m15.367345417s) [noise=true , address=77.90.31.14:4661 , piece_id=QIAWNRNZD7IT2LFSIMJF62KG5ZJGHVVDMBSMUGEGMWJ5PBGJGGRQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: <nil> Dec 30 09:10:15.496287490 (593h41m15.367370535s) [seg_pos={0 0}, upload=520457]: Segment reached optimal threshold of 13 pieces. Dec 30 09:10:15.496545301 (593h41m15.367628336s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496554068 (593h41m15.367637113s) [noise=true , address=145.79.227.35:4482 , piece_id=Q3UIMUJASBKSFDDHB2ZTXM2FIZCMPGIN7BYMQJO2JR4BHXJD2YQQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496800638 (593h41m15.367883683s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): internal: context canceled Dec 30 09:10:15.496819944 (593h41m15.367902989s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ32TOQBIBPKQRLN6Z4T4CXDEJSZKZQIUFUSOAJRGZLAA7DPPZA, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496825865 (593h41m15.3679089s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 09:10:15.496833781 (593h41m15.367916816s) [noise=true , address=145.79.227.34:4906 , piece_id=ZSXNE7ETUYZP2XBM2L6HKJNVJNZPN3CVLH3RD234X5RTZ66OIJBQ, seg_pos={0 0}, upload=520457]: Long tail cancellation due to optimal threshold Dec 30 09:10:15.496852255 (593h41m15.367935301s) [seg_pos={0 0}, upload=520457]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:10:15.496963056 (593h41m15.368046091s) [upload=520457]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:10:15.497015326 (593h41m15.368098371s) [upload=520457]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:10:15.782201488 (593h41m15.653284533s) [upload=520457]: Done uploading segments... Dec 30 09:10:15.782220714 (593h41m15.653303749s) [upload=520457]: Upload finished. err: <nil> Duration: 30.00337504s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.............X....X.....X.........X...X.X....X.X...X........] duration: 4.676920189s trace: https://jaeger.dev.storj.io/trace/6c407bbbae47006e FAIL (2025-12-30 09:10:22 UTC) orbiter warning: context deadline exceeded Duration: 4.676955156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..X] duration: 30.002598274s trace: https://jaeger.dev.storj.io/trace/b1a0e5e34a7132a FAIL (2025-12-30 09:10:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 09:09:39.130855887 (593h40m39.001938922s) [upload=520453]: Starting upload Dec 30 09:09:39.130919378 (593h40m39.002002424s) [upload=520453]: Done starting upload Dec 30 09:09:39.130977418 (593h40m39.002060463s) [upload=520453]: Uploading segments... Dec 30 09:09:39.131587821 (593h40m39.002670856s) [upload=520453]: Got next segment. Inline: false Dec 30 09:09:39.131669266 (593h40m39.002752311s) [upload=520453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:09:39.131676210 (593h40m39.002759255s) [upload=520453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:09:39.376261380 (593h40m39.247344415s) [seg_pos={0 0}, upload=520453]: Begin upload segment... Dec 30 09:09:39.376312157 (593h40m39.247395192s) [seg_pos={0 0}, upload=520453]: Done begin upload segment. Dec 30 09:09:39.376352243 (593h40m39.247435289s) [noise=true , address=145.79.227.19:4077 , piece_id=ERDOY76J6FC3DM34OULMJJ2BPKGPY5SBZ36QVLK6AREQEVLY6GNQ, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.376382441 (593h40m39.247465496s) [noise=true , address=145.79.227.52:4446 , piece_id=K332ZXG6ENQEBNHEQHW3HRN4CCU62TCV2C3VCDI7GDXKB6O2EXXA, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.376967023 (593h40m39.248050079s) [noise=true , address=45.91.116.14:4190 , piece_id=27AARIXY24JRKQOVWSUIIIDDUM6H6OF4BBSV2ZXNG6PKJNHBH2VQ, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.377381593 (593h40m39.248464638s) [noise=true , address=77.90.31.17:4770 , piece_id=QXQX4JTHL52H42PED4WRIZEOHUM2X3G47ZAQOIT3ON7THXGLULJQ, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.377778890 (593h40m39.248861935s) [noise=true , address=77.90.31.14:4663 , piece_id=ADSB7SA2NOZJT355GF4IC2XNKLUH5ULBY7YASKC3TGRELMIFPE6Q, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.378213938 (593h40m39.249296973s) [noise=true , address=145.79.227.46:4603 , piece_id=6Q7TLTEPRTIXRBGBZVT6P5TDXDALOOBSKHPKBT7WECR7IWOZNHDA, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.378607258 (593h40m39.249690303s) [noise=true , address=77.90.31.18:4805 , piece_id=TXATSZUISJDD4BJ3NBVBWRUFHEJZ4IWLPLAD62TGMGAHNMMSA3UA, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.379084325 (593h40m39.25016736s) [noise=true , address=145.79.227.13:4803 , piece_id=7HJMOGDMPPWCZ2LV74PSF56XQWJFWRJVSSQYP3XDNOGV4EPVOAKQ, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.379759171 (593h40m39.250842216s) [noise=true , address=145.79.227.16:4911 , piece_id=N6GOA4IXVQGR2EP2MNOCCKZS3UZYTG7HUHCHCZJUP7G3IH4QVRGQ, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.380273610 (593h40m39.251356645s) [noise=true , address=145.79.227.4:4186 , piece_id=YIA7NNRLQXCTZH7HZGV2YP6BNBF7E3AD37FE63K2Y3WMSQTT6ISA, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.380709821 (593h40m39.251792866s) [noise=true , address=77.90.31.16:4736 , piece_id=322RBMEDUVI2ELDWPV2KN7NI3U4GXLKGGSSCZ6KZYRB6AD4IYEZQ, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.381215865 (593h40m39.25229891s) [noise=true , address=145.79.227.49:4339 , piece_id=KXCFSEHS6WTCDXFYDEGQL26PZSW4IPN2XS3ECQ6A6XNFR562T4SQ, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.381666702 (593h40m39.252749748s) [noise=true , address=45.91.116.7:4012 , piece_id=QSMNKZWDOJEE73KUN6ES4ZYPYSXT2KA5RBLSOSLQLWIN2RODO7IA, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.382104286 (593h40m39.253187331s) [noise=true , address=145.79.227.27:4691 , piece_id=ZKQ6HAZQ7Z4WMGBHMUABHPOYMVTHYTXSE74U4QUJWPYWM6Z6XZBA, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.382660405 (593h40m39.25374345s) [noise=true , address=145.79.227.6:4257 , piece_id=ZYQSH3EVJDECI4HIZMORI55AAAUU23DJFNTOLJTOQR72WEOPF7BA, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:39.382988189 (593h40m39.254071224s) [noise=true , address=145.79.227.11:4732 , piece_id=MP35K3UPLRZSTIOKVBPHX5TABOAFSXE3IO5SPG33SB2KVG2LWTFQ, seg_pos={0 0}, upload=520453]: Uploading piece... Dec 30 09:09:41.087167548 (593h40m40.958250593s) [upload=520453]: Next returned nil segment Dec 30 09:09:41.087194289 (593h40m40.958277334s) [upload=520453]: Waiting for error group managing segments... Dec 30 09:09:41.368994144 (593h40m41.240077189s) [noise=true , address=145.79.227.13:4803 , piece_id=7HJMOGDMPPWCZ2LV74PSF56XQWJFWRJVSSQYP3XDNOGV4EPVOAKQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.370028152 (593h40m41.241111197s) [noise=true , address=145.79.227.52:4446 , piece_id=K332ZXG6ENQEBNHEQHW3HRN4CCU62TCV2C3VCDI7GDXKB6O2EXXA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.377031237 (593h40m41.248114272s) [noise=true , address=145.79.227.19:4077 , piece_id=ERDOY76J6FC3DM34OULMJJ2BPKGPY5SBZ36QVLK6AREQEVLY6GNQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.378687892 (593h40m41.249770927s) [noise=true , address=145.79.227.16:4911 , piece_id=N6GOA4IXVQGR2EP2MNOCCKZS3UZYTG7HUHCHCZJUP7G3IH4QVRGQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.382652120 (593h40m41.253735165s) [noise=true , address=145.79.227.46:4603 , piece_id=6Q7TLTEPRTIXRBGBZVT6P5TDXDALOOBSKHPKBT7WECR7IWOZNHDA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.385120671 (593h40m41.256203736s) [noise=true , address=145.79.227.4:4186 , piece_id=YIA7NNRLQXCTZH7HZGV2YP6BNBF7E3AD37FE63K2Y3WMSQTT6ISA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.388830875 (593h40m41.25991391s) [noise=true , address=145.79.227.6:4257 , piece_id=ZYQSH3EVJDECI4HIZMORI55AAAUU23DJFNTOLJTOQR72WEOPF7BA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.392115169 (593h40m41.263198204s) [noise=true , address=145.79.227.11:4732 , piece_id=MP35K3UPLRZSTIOKVBPHX5TABOAFSXE3IO5SPG33SB2KVG2LWTFQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.441332297 (593h40m41.312415332s) [noise=true , address=145.79.227.27:4691 , piece_id=ZKQ6HAZQ7Z4WMGBHMUABHPOYMVTHYTXSE74U4QUJWPYWM6Z6XZBA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.694832283 (593h40m41.565915328s) [noise=true , address=45.91.116.7:4012 , piece_id=QSMNKZWDOJEE73KUN6ES4ZYPYSXT2KA5RBLSOSLQLWIN2RODO7IA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.711756859 (593h40m41.582839895s) [noise=true , address=77.90.31.14:4663 , piece_id=ADSB7SA2NOZJT355GF4IC2XNKLUH5ULBY7YASKC3TGRELMIFPE6Q, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:41.977416800 (593h40m41.848499835s) [noise=true , address=77.90.31.17:4770 , piece_id=QXQX4JTHL52H42PED4WRIZEOHUM2X3G47ZAQOIT3ON7THXGLULJQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:42.218243134 (593h40m42.089326169s) [noise=true , address=77.90.31.18:4805 , piece_id=TXATSZUISJDD4BJ3NBVBWRUFHEJZ4IWLPLAD62TGMGAHNMMSA3UA, seg_pos={0 0}, upload=520453]: Done uploading piece. err: <nil> Dec 30 09:09:42.218260999 (593h40m42.089344034s) [seg_pos={0 0}, upload=520453]: Segment reached optimal threshold of 13 pieces. Dec 30 09:09:42.218450148 (593h40m42.089533193s) [noise=true , address=145.79.227.49:4339 , piece_id=KXCFSEHS6WTCDXFYDEGQL26PZSW4IPN2XS3ECQ6A6XNFR562T4SQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 09:09:42.218456089 (593h40m42.089539124s) [noise=true , address=145.79.227.49:4339 , piece_id=KXCFSEHS6WTCDXFYDEGQL26PZSW4IPN2XS3ECQ6A6XNFR562T4SQ, seg_pos={0 0}, upload=520453]: Long tail cancellation due to optimal threshold Dec 30 09:09:42.218676951 (593h40m42.089759986s) [noise=true , address=45.91.116.14:4190 , piece_id=27AARIXY24JRKQOVWSUIIIDDUM6H6OF4BBSV2ZXNG6PKJNHBH2VQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52578->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52578->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52578->45.91.116.14:4190: use of closed network connection Dec 30 09:09:42.218684174 (593h40m42.089767209s) [noise=true , address=45.91.116.14:4190 , piece_id=27AARIXY24JRKQOVWSUIIIDDUM6H6OF4BBSV2ZXNG6PKJNHBH2VQ, seg_pos={0 0}, upload=520453]: Long tail cancellation due to optimal threshold Dec 30 09:09:42.218815815 (593h40m42.08989886s) [noise=true , address=77.90.31.16:4736 , piece_id=322RBMEDUVI2ELDWPV2KN7NI3U4GXLKGGSSCZ6KZYRB6AD4IYEZQ, seg_pos={0 0}, upload=520453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 30 09:09:42.218822007 (593h40m42.089905042s) [noise=true , address=77.90.31.16:4736 , piece_id=322RBMEDUVI2ELDWPV2KN7NI3U4GXLKGGSSCZ6KZYRB6AD4IYEZQ, seg_pos={0 0}, upload=520453]: Long tail cancellation due to optimal threshold Dec 30 09:09:42.218836123 (593h40m42.089919158s) [seg_pos={0 0}, upload=520453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:09:42.218912538 (593h40m42.089995574s) [upload=520453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:09:42.218932998 (593h40m42.090016033s) [upload=520453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:09:42.508927415 (593h40m42.38001045s) [upload=520453]: Done uploading segments... Dec 30 09:09:42.508946511 (593h40m42.380029556s) [upload=520453]: Upload finished. err: <nil> Duration: 30.00262187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.....................X.......................] duration: 15.154987182s trace: https://jaeger.dev.storj.io/trace/e14e3f77cfefb30 FAIL (2025-12-30 09:09:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments... Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956 , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594 , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713 , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces. Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments... Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil> Duration: 15.155016067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..............X....X.....X.........X...X.X....X.X...X.......] duration: 5.001127755s trace: https://jaeger.dev.storj.io/trace/6c407bbbae47006e FAIL (2025-12-30 09:09:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001151349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...............X....X.....X.........X...X.X....X.X...X......] duration: 3.565165847s trace: https://jaeger.dev.storj.io/trace/6c407bbbae47006e FAIL (2025-12-30 09:09:08 UTC) orbiter warning: context deadline exceeded Duration: 3.565322465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.....................X......................] duration: 38.874158444s trace: https://jaeger.dev.storj.io/trace/e14e3f77cfefb30 FAIL (2025-12-30 09:09:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807118843 (593h21m35.678201878s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807337019 (593h21m35.678420054s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807436869 (593h21m35.678519904s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.807815018 (593h21m35.678898053s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808170957 (593h21m35.679253992s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808550199 (593h21m35.679633234s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808897170 (593h21m35.679980205s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.808938168 (593h21m35.680021213s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.809241615 (593h21m35.68032466s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.810184420 (593h21m35.681267455s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.813924442 (593h21m35.685007477s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.815805704 (593h21m35.68688878s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.817128213 (593h21m35.688211248s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:35.818412769 (593h21m35.689495814s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Uploading piece... Dec 30 08:50:37.246488707 (593h21m37.117571742s) [upload=520173]: Next returned nil segment Dec 30 08:50:37.246504797 (593h21m37.117587842s) [upload=520173]: Waiting for error group managing segments... Dec 30 08:50:37.617119537 (593h21m37.488202582s) [noise=true , address=64.49.13.27:4964 , piece_id=3IOD6GOR2JUQHZIWAQAEXBBGKJCBITIZE6MRANNS4TFGMIGN4LNA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.656276388 (593h21m37.527359423s) [noise=true , address=64.49.13.26:4956 , piece_id=SU2QROGRLIS4YQFE2RES3PH2S32BHVI3TXF7U2JGDYC3K366KNNQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.692596076 (593h21m37.563679111s) [noise=true , address=145.79.227.17:4987 , piece_id=26ZUGT6NIH4IOR6SKC6VWFZMRZNVOELETN7GGJG632QJWLKVRYYQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.733155218 (593h21m37.604238253s) [noise=true , address=145.79.227.29:4955 , piece_id=JCX2CSRZEP2PMGJPLGWBHV4UORVX3RE3ZOV656PY5RU4HF6ALH2A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.905397355 (593h21m37.77648039s) [noise=true , address=77.90.31.14:4683 , piece_id=2GLICJXR46KXKR2NWUQICK4FL42HHF6UAR4TXVFXR3NB32YQDUHA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.951372488 (593h21m37.822455523s) [noise=true , address=77.90.31.12:4594 , piece_id=NWJGVMQ7CHOREBNQ33IFVKBQUXDPU7RDS2RS4GJISEFYJCFTJA7A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:37.963800493 (593h21m37.834883529s) [noise=true , address=77.90.31.15:4713 , piece_id=SBSHVYFUZHPAVVRPF7CIH73RT7A3EDX7UTZM6CRSDM5XUBURA3AQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.149230773 (593h21m38.020313808s) [noise=true , address=145.79.227.40:4055 , piece_id=A7DPT5GGHF3ZRS3A4SWTBIQOGXP5NHFE23BHNGWYQSTOEJXD33QQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.180461882 (593h21m38.051544917s) [noise=true , address=45.91.116.13:4186 , piece_id=E5PRUUMJWUCSE33VQZ5M62GIOIRW4WMLZLRVIPWWEUVSNL6WAXUA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.195839091 (593h21m38.066922136s) [noise=true , address=77.90.31.21:4902 , piece_id=5DSA5INCNXIXC7V4RJX7DTRPWN7IV2IIXHS3UIERFMB2Z6QBUR3A, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.557140909 (593h21m38.428223954s) [noise=true , address=145.79.227.44:4202 , piece_id=SJGLBUIHJOVVRCJHYHIZ333TUZ7HCZWIJEWRMUP235YGCMHBFAZA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.564815133 (593h21m38.435898178s) [noise=true , address=193.149.164.143:4696 , piece_id=U26EFERLZTQ4635BUUF4ZHQ5AK4LV4EPYBYFK75HSGAPRJSQV3HA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680119209 (593h21m38.551202255s) [noise=true , address=145.79.227.30:4765 , piece_id=YHPLTJX7A7TEZ6YUDERQLEAIUAZXILETKG6RQI2WA2CPJIA3BQCQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680487441 (593h21m38.551570486s) [noise=true , address=145.79.227.34:4934 , piece_id=NRB3NAOI2GZKRBUU7EVVH3BD3W7YKTUY2JCNGUI6SCWGE6A2D5CQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: <nil> Dec 30 08:50:38.680509433 (593h21m38.551592468s) [seg_pos={0 0}, upload=520173]: Segment reached optimal threshold of 14 pieces. Dec 30 08:50:38.680771542 (593h21m38.551854577s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48684->64.49.13.169:4707: use of closed network connection Dec 30 08:50:38.680782382 (593h21m38.551865427s) [noise=true , address=64.49.13.169:4707 , piece_id=KMLWVT7YRKYSGOL76PSZQGXWCRDKOGFIN7CSKBWNM7KDVYN7XQPA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.680806418 (593h21m38.551889453s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 30 08:50:38.680814443 (593h21m38.551897488s) [noise=true , address=145.79.227.35:4496 , piece_id=4SUIGSUGSABL76I7ZMFXBFMFSJX3EUREZUPLNEBEHCMZIR2MZIJA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681009154 (593h21m38.5520922s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX): protocol: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57998->145.79.227.43:4170: use of closed network connection Dec 30 08:50:38.681014925 (593h21m38.55209797s) [noise=true , address=145.79.227.43:4170 , piece_id=OTJQZRRMCTAGA6TSNK3XRXVPFJZTD3NSDRPSSLH7F2EE6YTWCNBQ, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681264851 (593h21m38.552347886s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: context canceled; context canceled Dec 30 08:50:38.681273046 (593h21m38.552356081s) [noise=true , address=145.79.227.33:4881 , piece_id=Y533AVONHIRVWMP3IIK4W3T3OPYKOEVDAM5EF742HDJOEFJPB3YA, seg_pos={0 0}, upload=520173]: Long tail cancellation due to optimal threshold Dec 30 08:50:38.681288476 (593h21m38.552371511s) [seg_pos={0 0}, upload=520173]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:50:38.681367637 (593h21m38.552450672s) [upload=520173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:50:38.681372125 (593h21m38.55245517s) [upload=520173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:50:38.968146240 (593h21m38.839229275s) [upload=520173]: Done uploading segments... Dec 30 08:50:38.968170716 (593h21m38.839253762s) [upload=520173]: Upload finished. err: <nil> Duration: 38.874189311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX..] duration: 6.007206706s trace: https://jaeger.dev.storj.io/trace/5c53eea50bba27b FAIL (2025-12-30 09:09:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ) context deadline exceeded ======LOGS====== :08:10.690779532 (593h39m10.561862577s) [upload=520429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:08:10.690786285 (593h39m10.56186932s) [upload=520429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:08:10.935761318 (593h39m10.806844353s) [seg_pos={0 0}, upload=520429]: Begin upload segment... Dec 30 09:08:10.935816012 (593h39m10.806899047s) [seg_pos={0 0}, upload=520429]: Done begin upload segment. Dec 30 09:08:10.935848785 (593h39m10.80693182s) [noise=true , address=145.79.227.3:4150 , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.935945799 (593h39m10.807028865s) [noise=true , address=145.79.227.36:4517 , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.936630543 (593h39m10.807713588s) [noise=true , address=145.79.227.32:4836 , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.936760530 (593h39m10.807843566s) [noise=true , address=145.79.227.16:4909 , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937005657 (593h39m10.808088692s) [noise=true , address=145.79.227.45:4231 , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937221167 (593h39m10.808304202s) [noise=true , address=45.91.116.11:4117 , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937276663 (593h39m10.808359699s) [noise=true , address=145.79.227.20:4439 , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937529004 (593h39m10.808612039s) [noise=true , address=145.79.227.46:4601 , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937674771 (593h39m10.808757816s) [noise=true , address=45.91.116.9:4263 , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937824377 (593h39m10.808907412s) [noise=true , address=77.90.31.21:4877 , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938148073 (593h39m10.809231108s) [noise=true , address=145.79.227.14:4839 , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938240058 (593h39m10.809323103s) [noise=true , address=145.79.227.37:4567 , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938599753 (593h39m10.809682788s) [noise=true , address=145.79.227.50:4373 , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938780837 (593h39m10.809863883s) [noise=true , address=145.79.227.5:4221 , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938888933 (593h39m10.809971978s) [noise=true , address=77.90.31.15:4697 , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.939302892 (593h39m10.810385937s) [noise=true , address=145.79.227.17:4972 , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:12.644838984 (593h39m12.515922019s) [upload=520429]: Next returned nil segment Dec 30 09:08:12.644852830 (593h39m12.515935865s) [upload=520429]: Waiting for error group managing segments... Dec 30 09:08:12.929694426 (593h39m12.800777471s) [noise=true , address=145.79.227.17:4972 , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:12.933169093 (593h39m12.804252128s) [noise=true , address=145.79.227.37:4567 , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:12.944748601 (593h39m12.815831636s) [noise=true , address=145.79.227.3:4150 , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:12.951606620 (593h39m12.822689655s) [noise=true , address=145.79.227.20:4439 , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.011305357 (593h39m12.882388402s) [noise=true , address=145.79.227.32:4836 , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.311682716 (593h39m13.182765751s) [noise=true , address=45.91.116.11:4117 , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.878441738 (593h39m13.749524783s) [noise=true , address=77.90.31.21:4877 , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.931694179 (593h39m13.802777225s) [noise=true , address=45.91.116.9:4263 , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.956336944 (593h39m13.827419989s) [noise=true , address=77.90.31.15:4697 , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:16.396503654 (593h39m16.267586699s) [noise=true , address=145.79.227.5:4221 , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:17.359300108 (593h39m17.230383143s) [noise=true , address=145.79.227.14:4839 , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:18.044962994 (593h39m17.916046029s) [noise=true , address=145.79.227.36:4517 , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:19.793143977 (593h39m19.664227032s) [noise=true , address=145.79.227.16:4909 , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:19.793166420 (593h39m19.664249466s) [seg_pos={0 0}, upload=520429]: Segment reached optimal threshold of 13 pieces. Dec 30 09:08:19.793446243 (593h39m19.664529288s) [noise=true , address=145.79.227.50:4373 , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection Dec 30 09:08:19.793453858 (593h39m19.664536903s) [noise=true , address=145.79.227.50:4373 , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold Dec 30 09:08:19.793651063 (593h39m19.664734098s) [noise=true , address=145.79.227.45:4231 , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: read tcp 188.166.225.170:42626->145.79.227.45:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 09:08:19.793666532 (593h39m19.664749567s) [noise=true , address=145.79.227.45:4231 , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold Dec 30 09:08:19.793697882 (593h39m19.664780917s) [noise=true , address=145.79.227.46:4601 , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection Dec 30 09:08:19.793706007 (593h39m19.664789042s) [noise=true , address=145.79.227.46:4601 , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold Dec 30 09:08:19.793725735 (593h39m19.66480877s) [seg_pos={0 0}, upload=520429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:08:19.793825996 (593h39m19.664909031s) [upload=520429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:08:19.793830825 (593h39m19.664913861s) [upload=520429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:08:20.078348705 (593h39m19.94943174s) [upload=520429]: Done uploading segments... Dec 30 09:08:20.078371338 (593h39m19.949454383s) [upload=520429]: Upload finished. err: <nil> Duration: 6.007235981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.X..X.XXX...X.X.X.X.........X.X.X.X......X....X...X..XX.] duration: 5.851487633s trace: https://jaeger.dev.storj.io/trace/5c53eea50bba27b FAIL (2025-12-30 09:08:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ) context deadline exceeded ======LOGS====== :08:10.690779532 (593h39m10.561862577s) [upload=520429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 09:08:10.690786285 (593h39m10.56186932s) [upload=520429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 09:08:10.935761318 (593h39m10.806844353s) [seg_pos={0 0}, upload=520429]: Begin upload segment... Dec 30 09:08:10.935816012 (593h39m10.806899047s) [seg_pos={0 0}, upload=520429]: Done begin upload segment. Dec 30 09:08:10.935848785 (593h39m10.80693182s) [noise=true , address=145.79.227.3:4150 , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.935945799 (593h39m10.807028865s) [noise=true , address=145.79.227.36:4517 , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.936630543 (593h39m10.807713588s) [noise=true , address=145.79.227.32:4836 , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.936760530 (593h39m10.807843566s) [noise=true , address=145.79.227.16:4909 , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937005657 (593h39m10.808088692s) [noise=true , address=145.79.227.45:4231 , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937221167 (593h39m10.808304202s) [noise=true , address=45.91.116.11:4117 , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937276663 (593h39m10.808359699s) [noise=true , address=145.79.227.20:4439 , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937529004 (593h39m10.808612039s) [noise=true , address=145.79.227.46:4601 , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937674771 (593h39m10.808757816s) [noise=true , address=45.91.116.9:4263 , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.937824377 (593h39m10.808907412s) [noise=true , address=77.90.31.21:4877 , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938148073 (593h39m10.809231108s) [noise=true , address=145.79.227.14:4839 , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938240058 (593h39m10.809323103s) [noise=true , address=145.79.227.37:4567 , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938599753 (593h39m10.809682788s) [noise=true , address=145.79.227.50:4373 , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938780837 (593h39m10.809863883s) [noise=true , address=145.79.227.5:4221 , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.938888933 (593h39m10.809971978s) [noise=true , address=77.90.31.15:4697 , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:10.939302892 (593h39m10.810385937s) [noise=true , address=145.79.227.17:4972 , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Uploading piece... Dec 30 09:08:12.644838984 (593h39m12.515922019s) [upload=520429]: Next returned nil segment Dec 30 09:08:12.644852830 (593h39m12.515935865s) [upload=520429]: Waiting for error group managing segments... Dec 30 09:08:12.929694426 (593h39m12.800777471s) [noise=true , address=145.79.227.17:4972 , piece_id=QKTQ7SIAXZX3IN53JGYZOZHIH4HCACKUBIACCOHH2KCIVGKF3LGQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:12.933169093 (593h39m12.804252128s) [noise=true , address=145.79.227.37:4567 , piece_id=6FVKCZNDNYZCHIVUJW25OVT37OEDWSO3FESSMTV4EBYPRCFOX4KA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:12.944748601 (593h39m12.815831636s) [noise=true , address=145.79.227.3:4150 , piece_id=6FXZDTDUA63OH3KF2RRYEYITBCU2VTOUTLP3SIRUMK66AONUHN2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:12.951606620 (593h39m12.822689655s) [noise=true , address=145.79.227.20:4439 , piece_id=YJI5EX24G3JOLJB5W6VDWVVKCAX23JTK73OTQTY7MUU4NNZNPZ5A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.011305357 (593h39m12.882388402s) [noise=true , address=145.79.227.32:4836 , piece_id=E2275I3AP2JN7NBVXYRCQ3SETKIDOLSXCOLGLYGIAF5HNV2SSE2Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.311682716 (593h39m13.182765751s) [noise=true , address=45.91.116.11:4117 , piece_id=K26NAYLVUQK2LLOO6KJBD4WVGDT3GXUZDFRWT2VRXPPVMIVZRIZA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.878441738 (593h39m13.749524783s) [noise=true , address=77.90.31.21:4877 , piece_id=N4O7SUS7KTYTAPOQLMCNXIRALRSFSHIZX5TFO7ETBDJ73AM6FOFQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.931694179 (593h39m13.802777225s) [noise=true , address=45.91.116.9:4263 , piece_id=NH4YVLJTQDX63DUCWIK746A4D23IJGANT66MWT53F3GDBQOK7FOQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:13.956336944 (593h39m13.827419989s) [noise=true , address=77.90.31.15:4697 , piece_id=BU7UAFLMSCFKJLFL5LBNFFOMSLVZRSL5XDCWKOQ2SYQSIN5EMM2A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:16.396503654 (593h39m16.267586699s) [noise=true , address=145.79.227.5:4221 , piece_id=GCYEZ36UG4YW4XE67H6XABJVXJJ7BBA3UMPMRFWYJCIQZLCDGT6A, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:17.359300108 (593h39m17.230383143s) [noise=true , address=145.79.227.14:4839 , piece_id=BUSAN23JFFMOX7FLLVVYYSB7YCHAU66BWKBYEDNU2FGR7MYFEJ4Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:18.044962994 (593h39m17.916046029s) [noise=true , address=145.79.227.36:4517 , piece_id=CQXYBUWV7G327NGCMABHD2ZJENIOZM7UADXTFNZZIYQBFFBUOMQQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:19.793143977 (593h39m19.664227032s) [noise=true , address=145.79.227.16:4909 , piece_id=BKQ4K53QI3L2XAWSYCKMBUKLL4VAWKASZH5SLZOI5TI7YYYAZPUQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: <nil> Dec 30 09:08:19.793166420 (593h39m19.664249466s) [seg_pos={0 0}, upload=520429]: Segment reached optimal threshold of 13 pieces. Dec 30 09:08:19.793446243 (593h39m19.664529288s) [noise=true , address=145.79.227.50:4373 , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58086->145.79.227.50:4373: use of closed network connection Dec 30 09:08:19.793453858 (593h39m19.664536903s) [noise=true , address=145.79.227.50:4373 , piece_id=WOD24PMATL6QSCUXYIEJG5PBYD7YEBW44INXPSDSLJWLNZ5IG6AQ, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold Dec 30 09:08:19.793651063 (593h39m19.664734098s) [noise=true , address=145.79.227.45:4231 , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: read tcp 188.166.225.170:42626->145.79.227.45:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 09:08:19.793666532 (593h39m19.664749567s) [noise=true , address=145.79.227.45:4231 , piece_id=YINFMNYY53W5VOEGID24WZJKLQOBXXY5DAQREB4OIFUPV66VK3YA, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold Dec 30 09:08:19.793697882 (593h39m19.664780917s) [noise=true , address=145.79.227.46:4601 , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43118->145.79.227.46:4601: use of closed network connection Dec 30 09:08:19.793706007 (593h39m19.664789042s) [noise=true , address=145.79.227.46:4601 , piece_id=VS3VVQVRGS55U727PHPP4O6PQQCQ5WNZGN4M7W4RMPQJ7EGEI47Q, seg_pos={0 0}, upload=520429]: Long tail cancellation due to optimal threshold Dec 30 09:08:19.793725735 (593h39m19.66480877s) [seg_pos={0 0}, upload=520429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 09:08:19.793825996 (593h39m19.664909031s) [upload=520429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 09:08:19.793830825 (593h39m19.664913861s) [upload=520429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 09:08:20.078348705 (593h39m19.94943174s) [upload=520429]: Done uploading segments... Dec 30 09:08:20.078371338 (593h39m19.949454383s) [upload=520429]: Upload finished. err: <nil> Duration: 5.851502842s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)