Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.561326033s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:43:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.561344598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.............X.............] duration: 40.164812782s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:42:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 40.164838942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX......] duration: 5.874291094s trace: https://jaeger.dev.storj.io/trace/48e8aa9553656273 FAIL (2025-12-27 05:42:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded ======LOGS====== Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments... Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment... Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment. Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments... Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces. Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments... Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil> Duration: 5.874329718s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.587914753s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:41:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.587951504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.............X............] duration: 36.213006746s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 36.213035531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX.....] duration: 6.110578365s trace: https://jaeger.dev.storj.io/trace/48e8aa9553656273 FAIL (2025-12-27 05:41:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded ======LOGS====== Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments... Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment... Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment. Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments... Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces. Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments... Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil> Duration: 6.110609334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX....] duration: 15.016613941s trace: https://jaeger.dev.storj.io/trace/48e8aa9553656273 FAIL (2025-12-27 05:41:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded ======LOGS====== Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments... Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment... Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment. Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments... Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces. Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments... Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil> Duration: 15.016643487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.............X...........] duration: 34.000006612s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:41:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 34.000021661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.484115119s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:40:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48414123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..................................X.............X..........] duration: 38.882465477s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:40:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 38.882493701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX...] duration: 6.252727001s trace: https://jaeger.dev.storj.io/trace/48e8aa9553656273 FAIL (2025-12-27 05:40:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded ======LOGS====== Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments... Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment... Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment. Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments... Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces. Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments... Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil> Duration: 6.252755626s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX..] duration: 5.753114873s trace: https://jaeger.dev.storj.io/trace/48e8aa9553656273 FAIL (2025-12-27 05:39:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded ======LOGS====== Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments... Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment... Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment. Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments... Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces. Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments... Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil> Duration: 5.753141834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.403695136s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:39:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40372357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..................................X.............X.........] duration: 30.409790845s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:39:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 30.409811515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..................................X.............X........] duration: 37.037788088s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:39:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 37.03780448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX.] duration: 5.765558538s trace: https://jaeger.dev.storj.io/trace/48e8aa9553656273 FAIL (2025-12-27 05:39:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded ======LOGS====== Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments... Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment... Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment. Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments... Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces. Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments... Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil> Duration: 5.765576241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...XX] duration: 30.003245589s trace: https://jaeger.dev.storj.io/trace/48e8aa9553656273 FAIL (2025-12-27 05:39:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ) context deadline exceeded ======LOGS====== Dec 27 05:38:35.939286149 (518h9m35.810369184s) [upload=454602]: Starting upload Dec 27 05:38:35.939368898 (518h9m35.810451943s) [upload=454602]: Done starting upload Dec 27 05:38:35.939574269 (518h9m35.810657304s) [upload=454602]: Uploading segments... Dec 27 05:38:35.939633090 (518h9m35.810716135s) [upload=454602]: Got next segment. Inline: false Dec 27 05:38:35.939670862 (518h9m35.810753897s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:38:35.939676293 (518h9m35.810759328s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:38:36.187300652 (518h9m36.058383687s) [seg_pos={0 0}, upload=454602]: Begin upload segment... Dec 27 05:38:36.187371327 (518h9m36.058454362s) [seg_pos={0 0}, upload=454602]: Done begin upload segment. Dec 27 05:38:36.187457480 (518h9m36.058540515s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.187480464 (518h9m36.058563499s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188071139 (518h9m36.059154174s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188520785 (518h9m36.059603821s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.188940705 (518h9m36.06002374s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189385120 (518h9m36.060468156s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.189863171 (518h9m36.060946216s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191254420 (518h9m36.062337465s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.191856306 (518h9m36.062939351s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.193465010 (518h9m36.064548045s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.195509362 (518h9m36.066592397s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.197643477 (518h9m36.068726522s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.198529203 (518h9m36.069612238s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200184886 (518h9m36.071267921s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.200435623 (518h9m36.071518668s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:36.202086727 (518h9m36.073169772s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Uploading piece... Dec 27 05:38:37.895126995 (518h9m37.76621003s) [upload=454602]: Next returned nil segment Dec 27 05:38:37.895139619 (518h9m37.766222664s) [upload=454602]: Waiting for error group managing segments... Dec 27 05:38:38.182630231 (518h9m38.053713266s) [noise=true , address=145.79.227.48:4302 , piece_id=DUEDZCWH2IHGPD6OVOP6PJDRB3PTHHJG2SOZ3HM5WDBIQMLKE5FA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.190248887 (518h9m38.061331922s) [noise=true , address=145.79.227.31:4798 , piece_id=VKKWE7Q2R3ISUID2ZDYHGSDVN6GG2FTPTFOMDNPAPWNJNWHBPJZQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.195457305 (518h9m38.06654034s) [noise=true , address=145.79.227.12:4765 , piece_id=PJYB5ZPMFTZASZCUZM3HLQHOFKTZG2FTHPEYABBIQ55LGXCNNKGA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.196619137 (518h9m38.067702172s) [noise=true , address=145.79.227.45:4229 , piece_id=FKETKEONLZQVPFRFFHUGJ3METZFXAISXJ3FPZSPDX4TYQV5RFBIA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.208845419 (518h9m38.079928464s) [noise=true , address=145.79.227.39:4014 , piece_id=KI7UD46PVYGFZC3KCG2BXXQSIFI65XVKQTQNZSMBZX5D7I4CDY3Q, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.209110794 (518h9m38.080193839s) [noise=true , address=145.79.227.42:4122 , piece_id=XDPJ76EYUZN7ES254PW322VYBHXGEIVPW5VQ2JWTUBTBNVT5JW6A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.219635946 (518h9m38.090718991s) [noise=true , address=145.79.227.47:4267 , piece_id=XF2UZO27XCAPFBX2QLMO5VR3FTZI6UWMMBBZNOHOKA5HP3XADGCA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.224679620 (518h9m38.095762665s) [noise=true , address=145.79.227.15:4873 , piece_id=LCB2OKOWGJEXYCS4SVPLSGENS5OOSNILUGEL6ADHKRWRXX2TJNVA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.225072278 (518h9m38.096155323s) [noise=true , address=145.79.227.38:4977 , piece_id=YME2KRTHOUJAZVVDQG6B5RWKP6O44NFHQSZU5V42RQ75V2BQ2OAA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.242467491 (518h9m38.113550526s) [noise=true , address=145.79.227.52:4447 , piece_id=7ZG5ZFQUGHPOF7YKT3OE4NSQOIIOCNGNRI2HHHTF7GVOQPPVU47A, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.386106705 (518h9m38.25718975s) [noise=true , address=77.90.31.10:4519 , piece_id=ZYMKSZHGFNUF6QCNCDXB52CN5Q3P2IVBGVF56HGAHZ26VSB4EZOQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.451647625 (518h9m38.32273066s) [noise=true , address=45.91.116.11:4118 , piece_id=TIL3QRV2NJ7CYBIZ3BLZRQ33XXF54DV5G4HVBLIZARTSJOJQC4FQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526374669 (518h9m38.397457714s) [noise=true , address=45.91.116.9:4263 , piece_id=NPG4O3D7TPRKS4LENS3H346ZXZSAOPX66U5KM62QGUHU2OY3UCNA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: <nil> Dec 27 05:38:38.526396651 (518h9m38.397479686s) [seg_pos={0 0}, upload=454602]: Segment reached optimal threshold of 13 pieces. Dec 27 05:38:38.526579188 (518h9m38.397662224s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526585671 (518h9m38.397668706s) [noise=true , address=77.90.31.18:4808 , piece_id=5SSUUOUWLOUCUQKXKZCGYV436XI3B6JWKHPQIFWG6B6YSY3A4TPQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526635746 (518h9m38.397718781s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526644844 (518h9m38.397727879s) [noise=true , address=45.91.116.13:4156 , piece_id=H4ROEUGPTEMJLM7RWCBUO2SDOROBUBN66OCE3UCN33AIUDFXTLSA, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526949112 (518h9m38.398032157s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:38.526955756 (518h9m38.398038791s) [noise=true , address=77.90.31.21:4878 , piece_id=ZSQKKHCFCRUSIFYTHDGN6U4LEMLOKB5Z3PIZN2QCL72FULPUPBFQ, seg_pos={0 0}, upload=454602]: Long tail cancellation due to optimal threshold Dec 27 05:38:38.526984760 (518h9m38.398067795s) [seg_pos={0 0}, upload=454602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:38:38.527100581 (518h9m38.398183616s) [upload=454602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:38:38.527105681 (518h9m38.398188716s) [upload=454602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:38:38.827258623 (518h9m38.698341659s) [upload=454602]: Done uploading segments... Dec 27 05:38:38.827276427 (518h9m38.698359462s) [upload=454602]: Upload finished. err: <nil> Duration: 30.003257372s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.553004748s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:38:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553030796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..................................X.............X.......] duration: 41.492355808s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:38:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 41.492386175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...X] duration: 30.002673568s trace: https://jaeger.dev.storj.io/trace/3749de954066bfbb FAIL (2025-12-27 05:38:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 05:38:05.703447414 (518h9m5.57453046s) [upload=454595]: Starting upload Dec 27 05:38:05.703548487 (518h9m5.574631532s) [upload=454595]: Done starting upload Dec 27 05:38:05.703597411 (518h9m5.574680466s) [upload=454595]: Uploading segments... Dec 27 05:38:05.703835654 (518h9m5.574918699s) [upload=454595]: Got next segment. Inline: false Dec 27 05:38:05.703878837 (518h9m5.574961882s) [upload=454595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:38:05.703885018 (518h9m5.574968063s) [upload=454595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:38:05.956331880 (518h9m5.827414925s) [seg_pos={0 0}, upload=454595]: Begin upload segment... Dec 27 05:38:05.956389159 (518h9m5.827472194s) [seg_pos={0 0}, upload=454595]: Done begin upload segment. Dec 27 05:38:05.956421671 (518h9m5.827504706s) [noise=true , address=145.79.227.42:4121 , piece_id=4WNRFOLHFPFPY6AXZC26KABO3A6ZLRDXMZ5OITARXEUS3P2CSJOA, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.956432692 (518h9m5.827515777s) [noise=true , address=145.79.227.49:4339 , piece_id=NNM5ZHZYZF6MHNWPXLB54KV7TSSANB7AAWVNWE3HFD7A4TOQW46A, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.956681916 (518h9m5.827764961s) [noise=true , address=145.79.227.16:4911 , piece_id=PWJTZHKKLOCLNQYEBQA3BXTMQE2C7XWTLPA7LFKHW2EGT3TUWSIQ, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.957270086 (518h9m5.828353121s) [noise=true , address=145.79.227.41:4087 , piece_id=4UMW3L3HEVGWTWSA3FGCZXPMPYKPBK2EPDOBQB7KQSWPF27EFP5Q, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.957596297 (518h9m5.828679342s) [noise=true , address=145.79.227.12:4765 , piece_id=2ZCUGTBIHHA2LFYJG2FOCQ4WOOOCHD5WAYIFLJ3ASXRMR37BATSQ, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.958055140 (518h9m5.829138176s) [noise=true , address=145.79.227.37:4566 , piece_id=LFOMJWVLCQZWBDLMOKOKHPXRJMOLPEJCL4ALCK5D3GXRQTCCOCOQ, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.958407011 (518h9m5.829490056s) [noise=true , address=77.90.31.19:4841 , piece_id=GE3FNE5OTBBSY3GCYY5LN52T46WA3IDIOF2VLJVYKOGKJATO5E7A, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.958735286 (518h9m5.829818321s) [noise=true , address=45.91.116.9:4261 , piece_id=IJRKJSZMRIXALRA6R4MI65YYVAVOBQXMQ5KCQUUCA25WSC7A5TYA, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.959194390 (518h9m5.830277485s) [noise=true , address=45.91.116.4:4937 , piece_id=2WYWBMHLXKEG7X7BWRU6W3KPLG25BLAJHRXOXWT4DB2Y7HY7P5MA, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.959545358 (518h9m5.830628403s) [noise=true , address=145.79.227.39:4015 , piece_id=7HDU27GQTZRD2YPAXQ7GAUWJMDF5KI5UBTE6MJB23ETJEBRHI5NA, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.959697929 (518h9m5.830780975s) [noise=true , address=45.91.116.13:4155 , piece_id=JMBJKL5EWIYDGNHGHEK5ZDPUTW6PES64RPOCGYHO6Y6L3KHLJTIA, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.960014693 (518h9m5.831097728s) [noise=true , address=145.79.227.26:4654 , piece_id=JBKE7EQ4QJBFSRLZW2SNO5BQPE4ONRNZHAB3BQM4XNQ3MLV5U2CQ, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.960316026 (518h9m5.831399061s) [noise=true , address=145.79.227.48:4303 , piece_id=6YVDDD3A6QPVYNTQ4U3GGMFZDEYRJW3WNJZS6HJYXJSPP6EWCQ2Q, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.960656354 (518h9m5.831739389s) [noise=true , address=145.79.227.28:4725 , piece_id=23X5AKKFKS6ZIONIHKYEXEHV64WLQVJILKL6D7QULBU2TYB2MCWA, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.960691090 (518h9m5.831774155s) [noise=true , address=145.79.227.2:4113 , piece_id=Y3TEXMHMU3T6TJTZ65NPA52BYSB4DYUKY3SXCC4SWJWTUXRZ3TZQ, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:05.960992574 (518h9m5.83207562s) [noise=true , address=145.79.227.47:4266 , piece_id=HOGSCKCVNHO45LI6CLVMCWQDD3LYF3EB6GWNY5WQCEKADF6ERQQA, seg_pos={0 0}, upload=454595]: Uploading piece... Dec 27 05:38:07.569690160 (518h9m7.440773195s) [upload=454595]: Next returned nil segment Dec 27 05:38:07.569703725 (518h9m7.44078676s) [upload=454595]: Waiting for error group managing segments... Dec 27 05:38:07.955580530 (518h9m7.826663575s) [noise=true , address=145.79.227.39:4015 , piece_id=7HDU27GQTZRD2YPAXQ7GAUWJMDF5KI5UBTE6MJB23ETJEBRHI5NA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.958978952 (518h9m7.830061987s) [noise=true , address=145.79.227.12:4765 , piece_id=2ZCUGTBIHHA2LFYJG2FOCQ4WOOOCHD5WAYIFLJ3ASXRMR37BATSQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.965378548 (518h9m7.836461593s) [noise=true , address=145.79.227.41:4087 , piece_id=4UMW3L3HEVGWTWSA3FGCZXPMPYKPBK2EPDOBQB7KQSWPF27EFP5Q, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.966018997 (518h9m7.837102032s) [noise=true , address=145.79.227.48:4303 , piece_id=6YVDDD3A6QPVYNTQ4U3GGMFZDEYRJW3WNJZS6HJYXJSPP6EWCQ2Q, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.966844770 (518h9m7.837927805s) [noise=true , address=145.79.227.28:4725 , piece_id=23X5AKKFKS6ZIONIHKYEXEHV64WLQVJILKL6D7QULBU2TYB2MCWA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.974160039 (518h9m7.845243084s) [noise=true , address=145.79.227.26:4654 , piece_id=JBKE7EQ4QJBFSRLZW2SNO5BQPE4ONRNZHAB3BQM4XNQ3MLV5U2CQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.994572978 (518h9m7.865656014s) [noise=true , address=145.79.227.42:4121 , piece_id=4WNRFOLHFPFPY6AXZC26KABO3A6ZLRDXMZ5OITARXEUS3P2CSJOA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.995578794 (518h9m7.866661839s) [noise=true , address=145.79.227.37:4566 , piece_id=LFOMJWVLCQZWBDLMOKOKHPXRJMOLPEJCL4ALCK5D3GXRQTCCOCOQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.996024162 (518h9m7.867107197s) [noise=true , address=145.79.227.16:4911 , piece_id=PWJTZHKKLOCLNQYEBQA3BXTMQE2C7XWTLPA7LFKHW2EGT3TUWSIQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.996478637 (518h9m7.867561672s) [noise=true , address=145.79.227.2:4113 , piece_id=Y3TEXMHMU3T6TJTZ65NPA52BYSB4DYUKY3SXCC4SWJWTUXRZ3TZQ, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:07.996834214 (518h9m7.867917249s) [noise=true , address=145.79.227.47:4266 , piece_id=HOGSCKCVNHO45LI6CLVMCWQDD3LYF3EB6GWNY5WQCEKADF6ERQQA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:08.000784727 (518h9m7.871867762s) [noise=true , address=145.79.227.49:4339 , piece_id=NNM5ZHZYZF6MHNWPXLB54KV7TSSANB7AAWVNWE3HFD7A4TOQW46A, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:08.197039069 (518h9m8.068122114s) [noise=true , address=45.91.116.13:4155 , piece_id=JMBJKL5EWIYDGNHGHEK5ZDPUTW6PES64RPOCGYHO6Y6L3KHLJTIA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:08.197063465 (518h9m8.06814651s) [seg_pos={0 0}, upload=454595]: Segment reached optimal threshold of 13 pieces. Dec 27 05:38:08.197356182 (518h9m8.068439227s) [noise=true , address=45.91.116.4:4937 , piece_id=2WYWBMHLXKEG7X7BWRU6W3KPLG25BLAJHRXOXWT4DB2Y7HY7P5MA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:08.197365490 (518h9m8.068448535s) [noise=true , address=45.91.116.4:4937 , piece_id=2WYWBMHLXKEG7X7BWRU6W3KPLG25BLAJHRXOXWT4DB2Y7HY7P5MA, seg_pos={0 0}, upload=454595]: Long tail cancellation due to optimal threshold Dec 27 05:38:08.197781532 (518h9m8.068864577s) [noise=true , address=45.91.116.9:4261 , piece_id=IJRKJSZMRIXALRA6R4MI65YYVAVOBQXMQ5KCQUUCA25WSC7A5TYA, seg_pos={0 0}, upload=454595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 05:38:08.197793295 (518h9m8.06887634s) [noise=true , address=45.91.116.9:4261 , piece_id=IJRKJSZMRIXALRA6R4MI65YYVAVOBQXMQ5KCQUUCA25WSC7A5TYA, seg_pos={0 0}, upload=454595]: Long tail cancellation due to optimal threshold Dec 27 05:38:08.198939889 (518h9m8.070022924s) [noise=true , address=77.90.31.19:4841 , piece_id=GE3FNE5OTBBSY3GCYY5LN52T46WA3IDIOF2VLJVYKOGKJATO5E7A, seg_pos={0 0}, upload=454595]: Done uploading piece. err: <nil> Dec 27 05:38:08.198987489 (518h9m8.070070524s) [seg_pos={0 0}, upload=454595]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 27 05:38:08.199114822 (518h9m8.070197877s) [upload=454595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:38:08.199122797 (518h9m8.070205832s) [upload=454595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:38:08.488019957 (518h9m8.359102992s) [upload=454595]: Done uploading segments... Dec 27 05:38:08.488041948 (518h9m8.359124993s) [upload=454595]: Upload finished. err: <nil> Duration: 30.002699839s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.483796525s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:38:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.483824547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.592469886s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:38:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592497888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..................................X.............X......] duration: 35.426153454s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:37:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 35.426168684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX...] duration: 5.934749357s trace: https://jaeger.dev.storj.io/trace/4170035c95ee3ad0 FAIL (2025-12-27 05:37:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 05:34:48.380482691 (518h5m48.251565726s) [upload=454544]: Starting upload Dec 27 05:34:48.380562924 (518h5m48.25164596s) [upload=454544]: Done starting upload Dec 27 05:34:48.380673735 (518h5m48.25175678s) [upload=454544]: Uploading segments... Dec 27 05:34:48.380782292 (518h5m48.251865337s) [upload=454544]: Got next segment. Inline: false Dec 27 05:34:48.380815305 (518h5m48.25189835s) [upload=454544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:34:48.380820384 (518h5m48.251903419s) [upload=454544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:34:48.626473509 (518h5m48.497556555s) [seg_pos={0 0}, upload=454544]: Begin upload segment... Dec 27 05:34:48.626528334 (518h5m48.497611369s) [seg_pos={0 0}, upload=454544]: Done begin upload segment. Dec 27 05:34:48.626556177 (518h5m48.497639212s) [noise=true , address=145.79.227.40:4050 , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.628331186 (518h5m48.499414232s) [noise=true , address=145.79.227.5:4221 , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.628728574 (518h5m48.499811619s) [noise=true , address=145.79.227.49:4338 , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629099119 (518h5m48.500182154s) [noise=true , address=145.79.227.41:4086 , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629419209 (518h5m48.500502244s) [noise=true , address=145.79.227.9:4366 , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629760218 (518h5m48.500843253s) [noise=true , address=45.91.116.6:4973 , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.630136835 (518h5m48.50121988s) [noise=true , address=145.79.227.37:4566 , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.631218024 (518h5m48.502301069s) [noise=true , address=145.79.227.42:4122 , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.632043446 (518h5m48.503126491s) [noise=true , address=145.79.227.48:4303 , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.632670550 (518h5m48.503753585s) [noise=true , address=45.91.116.4:4940 , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.634500525 (518h5m48.50558357s) [noise=true , address=145.79.227.2:4114 , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.635139160 (518h5m48.506222206s) [noise=true , address=45.91.116.13:4154 , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.635823263 (518h5m48.506906309s) [noise=true , address=145.79.227.36:4518 , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.636522755 (518h5m48.50760579s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.637098070 (518h5m48.508181105s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.637850353 (518h5m48.508933388s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:50.260369707 (518h5m50.131452752s) [upload=454544]: Next returned nil segment Dec 27 05:34:50.260382743 (518h5m50.131465778s) [upload=454544]: Waiting for error group managing segments... Dec 27 05:34:50.639212409 (518h5m50.510295444s) [noise=true , address=145.79.227.5:4221 , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.640973903 (518h5m50.512056938s) [noise=true , address=145.79.227.36:4518 , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.641271821 (518h5m50.512354866s) [noise=true , address=145.79.227.9:4366 , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.641696529 (518h5m50.512779564s) [noise=true , address=145.79.227.41:4086 , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.642399538 (518h5m50.513482583s) [noise=true , address=145.79.227.2:4114 , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.644172935 (518h5m50.51525597s) [noise=true , address=145.79.227.49:4338 , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.651155731 (518h5m50.522238776s) [noise=true , address=145.79.227.40:4050 , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.656354861 (518h5m50.527437896s) [noise=true , address=145.79.227.48:4303 , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.659760366 (518h5m50.530843411s) [noise=true , address=145.79.227.37:4566 , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.672522708 (518h5m50.543605743s) [noise=true , address=145.79.227.42:4122 , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.908330215 (518h5m50.77941326s) [noise=true , address=45.91.116.13:4154 , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.000300392 (518h5m50.871383427s) [noise=true , address=45.91.116.4:4940 , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.012693621 (518h5m50.883776676s) [noise=true , address=45.91.116.6:4973 , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.012717165 (518h5m50.88380021s) [seg_pos={0 0}, upload=454544]: Segment reached optimal threshold of 13 pieces. Dec 27 05:34:51.012957062 (518h5m50.884040107s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.012968935 (518h5m50.884051981s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013095285 (518h5m50.884178331s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.013113350 (518h5m50.884196395s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013296098 (518h5m50.884379143s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.013303251 (518h5m50.884386287s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013325995 (518h5m50.88440903s) [seg_pos={0 0}, upload=454544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:34:51.013423311 (518h5m50.884506346s) [upload=454544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:34:51.013492492 (518h5m50.884575537s) [upload=454544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:34:51.301485099 (518h5m51.172568134s) [upload=454544]: Done uploading segments... Dec 27 05:34:51.301504687 (518h5m51.172587732s) [upload=454544]: Upload finished. err: <nil> Duration: 5.934779595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..................................X.............X.....] duration: 48.607477528s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:37:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 48.607494841s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.598799941s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:36:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.598867119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX..] duration: 6.11886824s trace: https://jaeger.dev.storj.io/trace/4170035c95ee3ad0 FAIL (2025-12-27 05:36:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 05:34:48.380482691 (518h5m48.251565726s) [upload=454544]: Starting upload Dec 27 05:34:48.380562924 (518h5m48.25164596s) [upload=454544]: Done starting upload Dec 27 05:34:48.380673735 (518h5m48.25175678s) [upload=454544]: Uploading segments... Dec 27 05:34:48.380782292 (518h5m48.251865337s) [upload=454544]: Got next segment. Inline: false Dec 27 05:34:48.380815305 (518h5m48.25189835s) [upload=454544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:34:48.380820384 (518h5m48.251903419s) [upload=454544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:34:48.626473509 (518h5m48.497556555s) [seg_pos={0 0}, upload=454544]: Begin upload segment... Dec 27 05:34:48.626528334 (518h5m48.497611369s) [seg_pos={0 0}, upload=454544]: Done begin upload segment. Dec 27 05:34:48.626556177 (518h5m48.497639212s) [noise=true , address=145.79.227.40:4050 , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.628331186 (518h5m48.499414232s) [noise=true , address=145.79.227.5:4221 , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.628728574 (518h5m48.499811619s) [noise=true , address=145.79.227.49:4338 , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629099119 (518h5m48.500182154s) [noise=true , address=145.79.227.41:4086 , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629419209 (518h5m48.500502244s) [noise=true , address=145.79.227.9:4366 , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629760218 (518h5m48.500843253s) [noise=true , address=45.91.116.6:4973 , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.630136835 (518h5m48.50121988s) [noise=true , address=145.79.227.37:4566 , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.631218024 (518h5m48.502301069s) [noise=true , address=145.79.227.42:4122 , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.632043446 (518h5m48.503126491s) [noise=true , address=145.79.227.48:4303 , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.632670550 (518h5m48.503753585s) [noise=true , address=45.91.116.4:4940 , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.634500525 (518h5m48.50558357s) [noise=true , address=145.79.227.2:4114 , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.635139160 (518h5m48.506222206s) [noise=true , address=45.91.116.13:4154 , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.635823263 (518h5m48.506906309s) [noise=true , address=145.79.227.36:4518 , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.636522755 (518h5m48.50760579s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.637098070 (518h5m48.508181105s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.637850353 (518h5m48.508933388s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:50.260369707 (518h5m50.131452752s) [upload=454544]: Next returned nil segment Dec 27 05:34:50.260382743 (518h5m50.131465778s) [upload=454544]: Waiting for error group managing segments... Dec 27 05:34:50.639212409 (518h5m50.510295444s) [noise=true , address=145.79.227.5:4221 , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.640973903 (518h5m50.512056938s) [noise=true , address=145.79.227.36:4518 , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.641271821 (518h5m50.512354866s) [noise=true , address=145.79.227.9:4366 , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.641696529 (518h5m50.512779564s) [noise=true , address=145.79.227.41:4086 , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.642399538 (518h5m50.513482583s) [noise=true , address=145.79.227.2:4114 , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.644172935 (518h5m50.51525597s) [noise=true , address=145.79.227.49:4338 , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.651155731 (518h5m50.522238776s) [noise=true , address=145.79.227.40:4050 , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.656354861 (518h5m50.527437896s) [noise=true , address=145.79.227.48:4303 , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.659760366 (518h5m50.530843411s) [noise=true , address=145.79.227.37:4566 , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.672522708 (518h5m50.543605743s) [noise=true , address=145.79.227.42:4122 , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.908330215 (518h5m50.77941326s) [noise=true , address=45.91.116.13:4154 , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.000300392 (518h5m50.871383427s) [noise=true , address=45.91.116.4:4940 , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.012693621 (518h5m50.883776676s) [noise=true , address=45.91.116.6:4973 , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.012717165 (518h5m50.88380021s) [seg_pos={0 0}, upload=454544]: Segment reached optimal threshold of 13 pieces. Dec 27 05:34:51.012957062 (518h5m50.884040107s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.012968935 (518h5m50.884051981s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013095285 (518h5m50.884178331s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.013113350 (518h5m50.884196395s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013296098 (518h5m50.884379143s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.013303251 (518h5m50.884386287s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013325995 (518h5m50.88440903s) [seg_pos={0 0}, upload=454544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:34:51.013423311 (518h5m50.884506346s) [upload=454544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:34:51.013492492 (518h5m50.884575537s) [upload=454544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:34:51.301485099 (518h5m51.172568134s) [upload=454544]: Done uploading segments... Dec 27 05:34:51.301504687 (518h5m51.172587732s) [upload=454544]: Upload finished. err: <nil> Duration: 6.118897868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.688536384s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:36:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.688563224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..................................X.............X....] duration: 46.644974513s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:35:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 46.644994542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX.] duration: 5.811565903s trace: https://jaeger.dev.storj.io/trace/4170035c95ee3ad0 FAIL (2025-12-27 05:35:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 05:34:48.380482691 (518h5m48.251565726s) [upload=454544]: Starting upload Dec 27 05:34:48.380562924 (518h5m48.25164596s) [upload=454544]: Done starting upload Dec 27 05:34:48.380673735 (518h5m48.25175678s) [upload=454544]: Uploading segments... Dec 27 05:34:48.380782292 (518h5m48.251865337s) [upload=454544]: Got next segment. Inline: false Dec 27 05:34:48.380815305 (518h5m48.25189835s) [upload=454544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:34:48.380820384 (518h5m48.251903419s) [upload=454544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:34:48.626473509 (518h5m48.497556555s) [seg_pos={0 0}, upload=454544]: Begin upload segment... Dec 27 05:34:48.626528334 (518h5m48.497611369s) [seg_pos={0 0}, upload=454544]: Done begin upload segment. Dec 27 05:34:48.626556177 (518h5m48.497639212s) [noise=true , address=145.79.227.40:4050 , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.628331186 (518h5m48.499414232s) [noise=true , address=145.79.227.5:4221 , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.628728574 (518h5m48.499811619s) [noise=true , address=145.79.227.49:4338 , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629099119 (518h5m48.500182154s) [noise=true , address=145.79.227.41:4086 , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629419209 (518h5m48.500502244s) [noise=true , address=145.79.227.9:4366 , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629760218 (518h5m48.500843253s) [noise=true , address=45.91.116.6:4973 , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.630136835 (518h5m48.50121988s) [noise=true , address=145.79.227.37:4566 , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.631218024 (518h5m48.502301069s) [noise=true , address=145.79.227.42:4122 , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.632043446 (518h5m48.503126491s) [noise=true , address=145.79.227.48:4303 , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.632670550 (518h5m48.503753585s) [noise=true , address=45.91.116.4:4940 , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.634500525 (518h5m48.50558357s) [noise=true , address=145.79.227.2:4114 , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.635139160 (518h5m48.506222206s) [noise=true , address=45.91.116.13:4154 , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.635823263 (518h5m48.506906309s) [noise=true , address=145.79.227.36:4518 , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.636522755 (518h5m48.50760579s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.637098070 (518h5m48.508181105s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.637850353 (518h5m48.508933388s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:50.260369707 (518h5m50.131452752s) [upload=454544]: Next returned nil segment Dec 27 05:34:50.260382743 (518h5m50.131465778s) [upload=454544]: Waiting for error group managing segments... Dec 27 05:34:50.639212409 (518h5m50.510295444s) [noise=true , address=145.79.227.5:4221 , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.640973903 (518h5m50.512056938s) [noise=true , address=145.79.227.36:4518 , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.641271821 (518h5m50.512354866s) [noise=true , address=145.79.227.9:4366 , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.641696529 (518h5m50.512779564s) [noise=true , address=145.79.227.41:4086 , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.642399538 (518h5m50.513482583s) [noise=true , address=145.79.227.2:4114 , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.644172935 (518h5m50.51525597s) [noise=true , address=145.79.227.49:4338 , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.651155731 (518h5m50.522238776s) [noise=true , address=145.79.227.40:4050 , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.656354861 (518h5m50.527437896s) [noise=true , address=145.79.227.48:4303 , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.659760366 (518h5m50.530843411s) [noise=true , address=145.79.227.37:4566 , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.672522708 (518h5m50.543605743s) [noise=true , address=145.79.227.42:4122 , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.908330215 (518h5m50.77941326s) [noise=true , address=45.91.116.13:4154 , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.000300392 (518h5m50.871383427s) [noise=true , address=45.91.116.4:4940 , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.012693621 (518h5m50.883776676s) [noise=true , address=45.91.116.6:4973 , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.012717165 (518h5m50.88380021s) [seg_pos={0 0}, upload=454544]: Segment reached optimal threshold of 13 pieces. Dec 27 05:34:51.012957062 (518h5m50.884040107s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.012968935 (518h5m50.884051981s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013095285 (518h5m50.884178331s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.013113350 (518h5m50.884196395s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013296098 (518h5m50.884379143s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.013303251 (518h5m50.884386287s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013325995 (518h5m50.88440903s) [seg_pos={0 0}, upload=454544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:34:51.013423311 (518h5m50.884506346s) [upload=454544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:34:51.013492492 (518h5m50.884575537s) [upload=454544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:34:51.301485099 (518h5m51.172568134s) [upload=454544]: Done uploading segments... Dec 27 05:34:51.301504687 (518h5m51.172587732s) [upload=454544]: Upload finished. err: <nil> Duration: 5.811597655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.XX] duration: 30.003463762s trace: https://jaeger.dev.storj.io/trace/4170035c95ee3ad0 FAIL (2025-12-27 05:35:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 05:34:48.380482691 (518h5m48.251565726s) [upload=454544]: Starting upload Dec 27 05:34:48.380562924 (518h5m48.25164596s) [upload=454544]: Done starting upload Dec 27 05:34:48.380673735 (518h5m48.25175678s) [upload=454544]: Uploading segments... Dec 27 05:34:48.380782292 (518h5m48.251865337s) [upload=454544]: Got next segment. Inline: false Dec 27 05:34:48.380815305 (518h5m48.25189835s) [upload=454544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:34:48.380820384 (518h5m48.251903419s) [upload=454544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:34:48.626473509 (518h5m48.497556555s) [seg_pos={0 0}, upload=454544]: Begin upload segment... Dec 27 05:34:48.626528334 (518h5m48.497611369s) [seg_pos={0 0}, upload=454544]: Done begin upload segment. Dec 27 05:34:48.626556177 (518h5m48.497639212s) [noise=true , address=145.79.227.40:4050 , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.628331186 (518h5m48.499414232s) [noise=true , address=145.79.227.5:4221 , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.628728574 (518h5m48.499811619s) [noise=true , address=145.79.227.49:4338 , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629099119 (518h5m48.500182154s) [noise=true , address=145.79.227.41:4086 , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629419209 (518h5m48.500502244s) [noise=true , address=145.79.227.9:4366 , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.629760218 (518h5m48.500843253s) [noise=true , address=45.91.116.6:4973 , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.630136835 (518h5m48.50121988s) [noise=true , address=145.79.227.37:4566 , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.631218024 (518h5m48.502301069s) [noise=true , address=145.79.227.42:4122 , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.632043446 (518h5m48.503126491s) [noise=true , address=145.79.227.48:4303 , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.632670550 (518h5m48.503753585s) [noise=true , address=45.91.116.4:4940 , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.634500525 (518h5m48.50558357s) [noise=true , address=145.79.227.2:4114 , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.635139160 (518h5m48.506222206s) [noise=true , address=45.91.116.13:4154 , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.635823263 (518h5m48.506906309s) [noise=true , address=145.79.227.36:4518 , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.636522755 (518h5m48.50760579s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.637098070 (518h5m48.508181105s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:48.637850353 (518h5m48.508933388s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Uploading piece... Dec 27 05:34:50.260369707 (518h5m50.131452752s) [upload=454544]: Next returned nil segment Dec 27 05:34:50.260382743 (518h5m50.131465778s) [upload=454544]: Waiting for error group managing segments... Dec 27 05:34:50.639212409 (518h5m50.510295444s) [noise=true , address=145.79.227.5:4221 , piece_id=6DBGT5J6ZSUUEXSB7AGHQDQTXKXU56FQXQBKA2X3T4T4M3O5ZRLQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.640973903 (518h5m50.512056938s) [noise=true , address=145.79.227.36:4518 , piece_id=E7X76H4U35WM4FHCGTXEJV4ICVLLG7SDMGMJDUI6YGR3BVNJALKA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.641271821 (518h5m50.512354866s) [noise=true , address=145.79.227.9:4366 , piece_id=XLBACHABSBZGAKQVPJLNM4ETO6FEBHMEZ4KDIOTA7HMFS2DX7OEA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.641696529 (518h5m50.512779564s) [noise=true , address=145.79.227.41:4086 , piece_id=HJGQD2CTLDYPM3CAHXYMFSDXXMWOPRWRFQM3ARG7ECHW6NEHHRAQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.642399538 (518h5m50.513482583s) [noise=true , address=145.79.227.2:4114 , piece_id=ONFOZSZFDF3TUF4GO6RDUUQ5VOAQ4G2VZUWGN3FJKZ47PSKQR3RA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.644172935 (518h5m50.51525597s) [noise=true , address=145.79.227.49:4338 , piece_id=ZRWY47VF6XRDGQRITI3DFDQ7NDRJ2IT3AALENJYXC6BRC4GHMF4A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.651155731 (518h5m50.522238776s) [noise=true , address=145.79.227.40:4050 , piece_id=Y7TX5X6NOOABCQZTROFIZEJNDFKHHIZIINFYGQHVMEUHPYACNMJQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.656354861 (518h5m50.527437896s) [noise=true , address=145.79.227.48:4303 , piece_id=4IWWCJSJS7TXITOVQGXE55L2HEGEQ3FQO5PDS4IWAW5N6IXOCETA, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.659760366 (518h5m50.530843411s) [noise=true , address=145.79.227.37:4566 , piece_id=HHZ3NQF666UAO5TPYCGFI4CJTIQ3WDYMOLYAUMTMQX3YLO4VOA5A, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.672522708 (518h5m50.543605743s) [noise=true , address=145.79.227.42:4122 , piece_id=3CV3L5SFKWXT2DZHWHY3QVDUWLCPOBFWR3EOMKAZEEWOW6FNGGHQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:50.908330215 (518h5m50.77941326s) [noise=true , address=45.91.116.13:4154 , piece_id=QRZFSSX3UT6OTMHQXBYRZZQBHCX3TV2BND4ZK2MAQ4EBAYI2ILSQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.000300392 (518h5m50.871383427s) [noise=true , address=45.91.116.4:4940 , piece_id=DFZHCXK663VGSWGRMIQQGIBTO7QKOYWVLOKEN75V7KVDJ4UEJ6PQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.012693621 (518h5m50.883776676s) [noise=true , address=45.91.116.6:4973 , piece_id=YU2EOXICLU5EGUMWMORYRJSXJZNFB4NKAECEAWK6DBKKYWUWIVCQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: <nil> Dec 27 05:34:51.012717165 (518h5m50.88380021s) [seg_pos={0 0}, upload=454544]: Segment reached optimal threshold of 13 pieces. Dec 27 05:34:51.012957062 (518h5m50.884040107s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.012968935 (518h5m50.884051981s) [noise=true , address=45.91.116.5:4298 , piece_id=SPILD7MW5EPZ7DYKFODPOQOM35EOLXVOCJ7CD53Y54QWQ6VAN2DQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013095285 (518h5m50.884178331s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.013113350 (518h5m50.884196395s) [noise=true , address=45.91.116.9:4264 , piece_id=UFDGHP3UXNCM4GW5RFRYQA7A3YNXC2BTLA6HUMA4ZE7LX4DHKAXQ, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013296098 (518h5m50.884379143s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 05:34:51.013303251 (518h5m50.884386287s) [noise=true , address=77.90.31.10:4517 , piece_id=FKENRBI36FX6AQXKV57YFUB6PVH445KKHU6DEC3QEGUPJLHCXD4Q, seg_pos={0 0}, upload=454544]: Long tail cancellation due to optimal threshold Dec 27 05:34:51.013325995 (518h5m50.88440903s) [seg_pos={0 0}, upload=454544]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:34:51.013423311 (518h5m50.884506346s) [upload=454544]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:34:51.013492492 (518h5m50.884575537s) [upload=454544]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:34:51.301485099 (518h5m51.172568134s) [upload=454544]: Done uploading segments... Dec 27 05:34:51.301504687 (518h5m51.172587732s) [upload=454544]: Upload finished. err: <nil> Duration: 30.003487618s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.736629158s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:35:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.736721384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..................................X.............X...] duration: 30.822760533s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:35:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 30.822808064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.X] duration: 30.003754499s trace: https://jaeger.dev.storj.io/trace/358391e693211d77 FAIL (2025-12-27 05:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 18h4m39.055282373s) [upload=454528]: Uploading segments... Dec 27 05:33:39.184293426 (518h4m39.055376471s) [upload=454528]: Got next segment. Inline: false Dec 27 05:33:39.184324425 (518h4m39.05540746s) [upload=454528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:33:39.184329746 (518h4m39.055412781s) [upload=454528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:33:39.427398484 (518h4m39.298481529s) [seg_pos={0 0}, upload=454528]: Begin upload segment... Dec 27 05:33:39.427495479 (518h4m39.298578524s) [noise=true , address=77.90.31.16:4736 , piece_id=6FOL6RQSIWPZ3OIK4DJZHCCDIHQJL2RTPCUJBDYYR65YHW47CEQA, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.427532128 (518h4m39.298615173s) [noise=true , address=145.79.227.17:4971 , piece_id=CVINDY7NYGV6H3M4ZQPAQF2WUC5U6HRE7C423JA3BCXY73645QPQ, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.427580410 (518h4m39.298663445s) [noise=true , address=145.79.227.28:4726 , piece_id=VVXGIQM7536NIRBVY45HQMVKFEQG43MLVY4PY2URFIEHLMDEZXNQ, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.427985922 (518h4m39.299068957s) [noise=true , address=45.91.116.4:4937 , piece_id=M2PHNFQWR6H5G77NWNA2EI5OTM7YMYWHX3GG3KRRIFZR3MADDVLA, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.428333234 (518h4m39.299416269s) [noise=true , address=145.79.227.19:4079 , piece_id=73L4ELR46QNFUMVU5FKRCP5KG7XFDFR5KN4GIU3NJ3BM4OM23YHA, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.428661198 (518h4m39.299744243s) [noise=true , address=45.91.116.11:4120 , piece_id=BSIGGA2TUJGFGOOSXHKDROGO45IIHFY4WO5NW3SUMJGMJP3TQ2YQ, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.428986807 (518h4m39.300069852s) [noise=true , address=145.79.227.47:4266 , piece_id=APFBQBZRCFTN7IQRHWCNSJ2UIO7KBZWUNZ6OK3VBHFYOS5P233MA, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.429188772 (518h4m39.300271817s) [noise=true , address=77.90.31.10:4519 , piece_id=GX5E7HJR7HY6SFHEG3VYNMPH2UYUCB74N7BTFLWA5K2MA55BYD3A, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.429507739 (518h4m39.300590774s) [noise=true , address=45.91.116.6:4976 , piece_id=D6LXHFJ2L2KK24TSM3WFMJBKDIUSBPZPJDKPMFEBCV4KBOPL2STA, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.429810787 (518h4m39.300893832s) [noise=true , address=145.79.227.16:4911 , piece_id=477BYJYQI6LARZ4GPAJDDI4OERXHZUZQ2HT7R4K6INFNIVFPRUPQ, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.429922489 (518h4m39.301005524s) [seg_pos={0 0}, upload=454528]: Done begin upload segment. Dec 27 05:33:39.429933671 (518h4m39.301016706s) [noise=true , address=145.79.227.41:4085 , piece_id=JQWVF62DHPXPREDBZNEFSSXV27MMEOZMNWYVGTLJG4PSFCVVZ6AQ, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.430124915 (518h4m39.30120795s) [noise=true , address=45.91.116.5:4298 , piece_id=KOFWF43LC2KKV4PUDKYJGRWMENB4RGSGURBDKGUM2XT7VJTG4CDA, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.430450786 (518h4m39.301533821s) [noise=true , address=145.79.227.42:4122 , piece_id=N3QQGGSBU33LNE4OCL3ZIWI6A7J3WFMDQYKMI7SWNLNK5JHU4VNA, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.430841460 (518h4m39.301924505s) [noise=true , address=145.79.227.31:4797 , piece_id=DRL2GZX5WGHIDJ446X6T6DYGICUKS43OPE5D6PCIBEMAEOLXWYMQ, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.431241541 (518h4m39.302324576s) [noise=true , address=45.91.116.10:4082 , piece_id=EP7XNGMUAJVM5O5MML6ALMD2BRYQCNCJV3JGR22OHIU3NHCMKEAA, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:39.431628738 (518h4m39.302711773s) [noise=true , address=145.79.227.52:4446 , piece_id=N6MUPSYBDT45KZ2XRE2SRTL5X63X4HEAVOEXN4ZWEEJWRJ2SFLUQ, seg_pos={0 0}, upload=454528]: Uploading piece... Dec 27 05:33:41.047519762 (518h4m40.918602807s) [upload=454528]: Next returned nil segment Dec 27 05:33:41.047535763 (518h4m40.918618808s) [upload=454528]: Waiting for error group managing segments... Dec 27 05:33:41.432398983 (518h4m41.303482029s) [noise=true , address=145.79.227.42:4122 , piece_id=N3QQGGSBU33LNE4OCL3ZIWI6A7J3WFMDQYKMI7SWNLNK5JHU4VNA, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.434394423 (518h4m41.305477468s) [noise=true , address=145.79.227.19:4079 , piece_id=73L4ELR46QNFUMVU5FKRCP5KG7XFDFR5KN4GIU3NJ3BM4OM23YHA, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.435226177 (518h4m41.306309222s) [noise=true , address=145.79.227.17:4971 , piece_id=CVINDY7NYGV6H3M4ZQPAQF2WUC5U6HRE7C423JA3BCXY73645QPQ, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.439443808 (518h4m41.310526853s) [noise=true , address=145.79.227.31:4797 , piece_id=DRL2GZX5WGHIDJ446X6T6DYGICUKS43OPE5D6PCIBEMAEOLXWYMQ, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.449761805 (518h4m41.320844841s) [noise=true , address=145.79.227.16:4911 , piece_id=477BYJYQI6LARZ4GPAJDDI4OERXHZUZQ2HT7R4K6INFNIVFPRUPQ, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.452797968 (518h4m41.323881003s) [noise=true , address=145.79.227.47:4266 , piece_id=APFBQBZRCFTN7IQRHWCNSJ2UIO7KBZWUNZ6OK3VBHFYOS5P233MA, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.454147146 (518h4m41.325230191s) [noise=true , address=145.79.227.28:4726 , piece_id=VVXGIQM7536NIRBVY45HQMVKFEQG43MLVY4PY2URFIEHLMDEZXNQ, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.455150637 (518h4m41.326233682s) [noise=true , address=145.79.227.41:4085 , piece_id=JQWVF62DHPXPREDBZNEFSSXV27MMEOZMNWYVGTLJG4PSFCVVZ6AQ, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.477772764 (518h4m41.348855799s) [noise=true , address=145.79.227.52:4446 , piece_id=N6MUPSYBDT45KZ2XRE2SRTL5X63X4HEAVOEXN4ZWEEJWRJ2SFLUQ, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.672979323 (518h4m41.544062378s) [noise=true , address=45.91.116.6:4976 , piece_id=D6LXHFJ2L2KK24TSM3WFMJBKDIUSBPZPJDKPMFEBCV4KBOPL2STA, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.704801729 (518h4m41.575884774s) [noise=true , address=77.90.31.16:4736 , piece_id=6FOL6RQSIWPZ3OIK4DJZHCCDIHQJL2RTPCUJBDYYR65YHW47CEQA, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.810373357 (518h4m41.681456402s) [noise=true , address=45.91.116.11:4120 , piece_id=BSIGGA2TUJGFGOOSXHKDROGO45IIHFY4WO5NW3SUMJGMJP3TQ2YQ, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.816024978 (518h4m41.687108023s) [noise=true , address=45.91.116.10:4082 , piece_id=EP7XNGMUAJVM5O5MML6ALMD2BRYQCNCJV3JGR22OHIU3NHCMKEAA, seg_pos={0 0}, upload=454528]: Done uploading piece. err: <nil> Dec 27 05:33:41.816049605 (518h4m41.687132651s) [seg_pos={0 0}, upload=454528]: Segment reached optimal threshold of 13 pieces. Dec 27 05:33:41.816291055 (518h4m41.6873741s) [noise=true , address=77.90.31.10:4519 , piece_id=GX5E7HJR7HY6SFHEG3VYNMPH2UYUCB74N7BTFLWA5K2MA55BYD3A, seg_pos={0 0}, upload=454528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:43230->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:43230->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43230->77.90.31.10:4519: use of closed network connection Dec 27 05:33:41.816300653 (518h4m41.687383718s) [noise=true , address=77.90.31.10:4519 , piece_id=GX5E7HJR7HY6SFHEG3VYNMPH2UYUCB74N7BTFLWA5K2MA55BYD3A, seg_pos={0 0}, upload=454528]: Long tail cancellation due to optimal threshold Dec 27 05:33:41.816312896 (518h4m41.687395941s) [noise=true , address=45.91.116.5:4298 , piece_id=KOFWF43LC2KKV4PUDKYJGRWMENB4RGSGURBDKGUM2XT7VJTG4CDA, seg_pos={0 0}, upload=454528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:36112->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:36112->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36112->45.91.116.5:4298: use of closed network connection Dec 27 05:33:41.816325331 (518h4m41.687408376s) [noise=true , address=45.91.116.5:4298 , piece_id=KOFWF43LC2KKV4PUDKYJGRWMENB4RGSGURBDKGUM2XT7VJTG4CDA, seg_pos={0 0}, upload=454528]: Long tail cancellation due to optimal threshold Dec 27 05:33:41.816623277 (518h4m41.687706322s) [noise=true , address=45.91.116.4:4937 , piece_id=M2PHNFQWR6H5G77NWNA2EI5OTM7YMYWHX3GG3KRRIFZR3MADDVLA, seg_pos={0 0}, upload=454528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 05:33:41.816633627 (518h4m41.687716692s) [noise=true , address=45.91.116.4:4937 , piece_id=M2PHNFQWR6H5G77NWNA2EI5OTM7YMYWHX3GG3KRRIFZR3MADDVLA, seg_pos={0 0}, upload=454528]: Long tail cancellation due to optimal threshold Dec 27 05:33:41.816660378 (518h4m41.687743423s) [seg_pos={0 0}, upload=454528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:33:41.816745991 (518h4m41.687829037s) [upload=454528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:33:41.816752253 (518h4m41.687835318s) [upload=454528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:33:42.104184056 (518h4m41.975267111s) [upload=454528]: Done uploading segments... Dec 27 05:33:42.104205788 (518h4m41.975288833s) [upload=454528]: Upload finished. err: <nil> Duration: 30.003767434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..................................X.............X..] duration: 36.845479457s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 36.845507561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.535721164s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:34:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535752444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..................................X.............X.] duration: 33.395769797s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:33:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 33.395785688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X.] duration: 6.070693459s trace: https://jaeger.dev.storj.io/trace/2e2ac57bb762310 FAIL (2025-12-27 05:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: ZJLRO7VJZN3UANXPNFADTPCU2GN6L746CE3NTATHH33KF3J2HVRQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 6R7QOMQLWBBNR5FTZV5U242EYRRR4VR34ANJ5TYVZH4VOHL6WSHQ) context deadline exceeded ======LOGS====== Dec 27 05:32:01.185993362 (518h3m1.057076398s) [upload=454503]: Starting upload Dec 27 05:32:01.186051783 (518h3m1.057134818s) [upload=454503]: Done starting upload Dec 27 05:32:01.186233239 (518h3m1.057316274s) [upload=454503]: Uploading segments... Dec 27 05:32:01.186263216 (518h3m1.057346251s) [upload=454503]: Got next segment. Inline: false Dec 27 05:32:01.186295547 (518h3m1.057378592s) [upload=454503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:32:01.186299264 (518h3m1.057382309s) [upload=454503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:32:01.434283031 (518h3m1.305366086s) [seg_pos={0 0}, upload=454503]: Begin upload segment... Dec 27 05:32:01.434342294 (518h3m1.305425339s) [seg_pos={0 0}, upload=454503]: Done begin upload segment. Dec 27 05:32:01.434379726 (518h3m1.305462771s) [noise=true , address=45.91.116.7:4011 , piece_id=7NTRUPQSHZBB7SNSVM3KND5JPSZDYUYBU44D7SXDNNO4ER6XPUNQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.435624707 (518h3m1.306707782s) [noise=true , address=145.79.227.9:4366 , piece_id=C22XTURJLVKO7XWTDCWZYV2COIWR3Y3EBOGPZ67J6IV46Z6D5TKA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.436425772 (518h3m1.307508817s) [noise=true , address=45.91.116.13:4154 , piece_id=ZLSCTC2EMVNTTY3457LLJ2V7SGHSO6OIWN4XRCO7N3YHH4DFWHSA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.436929260 (518h3m1.308012305s) [noise=true , address=145.79.227.38:4979 , piece_id=DNZIH2DETCQLY7Q346RGBKZHKEZC7KVRUAYLQT3MNY5NRHU54WSQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.437494556 (518h3m1.308577611s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJLRO7VJZN3UANXPNFADTPCU2GN6L746CE3NTATHH33KF3J2HVRQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.439432878 (518h3m1.310515953s) [noise=true , address=145.79.227.28:4726 , piece_id=PT5IUNEPAUDKTULK4HN23AT4ZLROYYWH6ZUVXCBCKNRSZPTDUGNA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.445627834 (518h3m1.316710879s) [noise=true , address=145.79.227.51:4410 , piece_id=7J3NAEL44CC5WQNDBNVOCPTGTJLK6YK2MJYPJ2RB2EYXW6FHRR2Q, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.452350936 (518h3m1.323433981s) [noise=true , address=77.90.31.17:4770 , piece_id=5VBYLOFNVHX7GKLG7VQJ6UCFQHBOY3ONFOQODN57TPE34RLRK46Q, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.455498238 (518h3m1.326581293s) [noise=true , address=145.79.227.45:4229 , piece_id=KLUIGZWANQEXK6UYB5X3HUC2743KMYD7I42CVF6G37IGRRA6I56A, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.460578172 (518h3m1.331661227s) [noise=true , address=145.79.227.47:4267 , piece_id=M53IWGBGDHNUASN5UCJVHCKOV2SNPOCTHYIQR4IYL7IHB6R22AKA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.462177688 (518h3m1.333260733s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPGN6QD7GK5RBFJRHLMEL52VS677WM7Q7AWLUYNNKMF54TNCROZA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.466184448 (518h3m1.337267493s) [noise=true , address=45.91.116.10:4082 , piece_id=6R7QOMQLWBBNR5FTZV5U242EYRRR4VR34ANJ5TYVZH4VOHL6WSHQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.470217087 (518h3m1.341300142s) [noise=true , address=145.79.227.48:4301 , piece_id=DLCSM2GDNUZAIKRZMVHJZ7HGVGKGGX5F2D6TW2T4KCQE24MKUERQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.471269841 (518h3m1.342352876s) [noise=true , address=45.91.116.4:4939 , piece_id=S7DWNUJXPZPBPNWHY7GNAIKKT3WZJDOU3262NG7QE4NNH5L5BG6A, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.473323872 (518h3m1.344406917s) [noise=true , address=145.79.227.46:4602 , piece_id=67MSYE2PNVLEJVKVKHUO7MOOJZZFDSB3HAEY3GH3OSFPEYTF26GQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.476104126 (518h3m1.347187162s) [noise=true , address=145.79.227.42:4122 , piece_id=PWUEMAWAKH2B2AN5K3I3MGHHXXQGRPSWP2LK2N2A2KYDF5HVGDXQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:03.174272812 (518h3m3.045355857s) [upload=454503]: Next returned nil segment Dec 27 05:32:03.174289494 (518h3m3.045372539s) [upload=454503]: Waiting for error group managing segments... Dec 27 05:32:03.461700297 (518h3m3.332783342s) [noise=true , address=145.79.227.48:4301 , piece_id=DLCSM2GDNUZAIKRZMVHJZ7HGVGKGGX5F2D6TW2T4KCQE24MKUERQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.466944863 (518h3m3.338027909s) [noise=true , address=145.79.227.45:4229 , piece_id=KLUIGZWANQEXK6UYB5X3HUC2743KMYD7I42CVF6G37IGRRA6I56A, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.468292259 (518h3m3.339375294s) [noise=true , address=145.79.227.47:4267 , piece_id=M53IWGBGDHNUASN5UCJVHCKOV2SNPOCTHYIQR4IYL7IHB6R22AKA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.470285815 (518h3m3.34136886s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPGN6QD7GK5RBFJRHLMEL52VS677WM7Q7AWLUYNNKMF54TNCROZA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.473713602 (518h3m3.344796637s) [noise=true , address=145.79.227.42:4122 , piece_id=PWUEMAWAKH2B2AN5K3I3MGHHXXQGRPSWP2LK2N2A2KYDF5HVGDXQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.475040560 (518h3m3.346123605s) [noise=true , address=145.79.227.38:4979 , piece_id=DNZIH2DETCQLY7Q346RGBKZHKEZC7KVRUAYLQT3MNY5NRHU54WSQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.485090185 (518h3m3.35617323s) [noise=true , address=145.79.227.51:4410 , piece_id=7J3NAEL44CC5WQNDBNVOCPTGTJLK6YK2MJYPJ2RB2EYXW6FHRR2Q, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.485434701 (518h3m3.356517746s) [noise=true , address=145.79.227.9:4366 , piece_id=C22XTURJLVKO7XWTDCWZYV2COIWR3Y3EBOGPZ67J6IV46Z6D5TKA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.487168353 (518h3m3.358251398s) [noise=true , address=145.79.227.46:4602 , piece_id=67MSYE2PNVLEJVKVKHUO7MOOJZZFDSB3HAEY3GH3OSFPEYTF26GQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.497161140 (518h3m3.368244186s) [noise=true , address=145.79.227.28:4726 , piece_id=PT5IUNEPAUDKTULK4HN23AT4ZLROYYWH6ZUVXCBCKNRSZPTDUGNA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.698386150 (518h3m3.569469195s) [noise=true , address=45.91.116.10:4082 , piece_id=6R7QOMQLWBBNR5FTZV5U242EYRRR4VR34ANJ5TYVZH4VOHL6WSHQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.780186553 (518h3m3.651269598s) [noise=true , address=45.91.116.4:4939 , piece_id=S7DWNUJXPZPBPNWHY7GNAIKKT3WZJDOU3262NG7QE4NNH5L5BG6A, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.852027192 (518h3m3.723110237s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJLRO7VJZN3UANXPNFADTPCU2GN6L746CE3NTATHH33KF3J2HVRQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.852055425 (518h3m3.723138461s) [seg_pos={0 0}, upload=454503]: Segment reached optimal threshold of 13 pieces. Dec 27 05:32:03.852303117 (518h3m3.723386172s) [noise=true , address=45.91.116.7:4011 , piece_id=7NTRUPQSHZBB7SNSVM3KND5JPSZDYUYBU44D7SXDNNO4ER6XPUNQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:03.852312875 (518h3m3.72339592s) [noise=true , address=45.91.116.7:4011 , piece_id=7NTRUPQSHZBB7SNSVM3KND5JPSZDYUYBU44D7SXDNNO4ER6XPUNQ, seg_pos={0 0}, upload=454503]: Long tail cancellation due to optimal threshold Dec 27 05:32:03.852612236 (518h3m3.723695291s) [noise=true , address=45.91.116.13:4154 , piece_id=ZLSCTC2EMVNTTY3457LLJ2V7SGHSO6OIWN4XRCO7N3YHH4DFWHSA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:03.852621022 (518h3m3.723704067s) [noise=true , address=45.91.116.13:4154 , piece_id=ZLSCTC2EMVNTTY3457LLJ2V7SGHSO6OIWN4XRCO7N3YHH4DFWHSA, seg_pos={0 0}, upload=454503]: Long tail cancellation due to optimal threshold Dec 27 05:32:03.852793421 (518h3m3.723876456s) [noise=true , address=77.90.31.17:4770 , piece_id=5VBYLOFNVHX7GKLG7VQJ6UCFQHBOY3ONFOQODN57TPE34RLRK46Q, seg_pos={0 0}, upload=454503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:03.852802698 (518h3m3.723885733s) [noise=true , address=77.90.31.17:4770 , piece_id=5VBYLOFNVHX7GKLG7VQJ6UCFQHBOY3ONFOQODN57TPE34RLRK46Q, seg_pos={0 0}, upload=454503]: Long tail cancellation due to optimal threshold Dec 27 05:32:03.852827646 (518h3m3.723910691s) [seg_pos={0 0}, upload=454503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:32:03.852924460 (518h3m3.724007495s) [upload=454503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:03.852932045 (518h3m3.72401509s) [upload=454503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:04.152690630 (518h3m4.023773675s) [upload=454503]: Done uploading segments... Dec 27 05:32:04.152715217 (518h3m4.023798272s) [upload=454503]: Upload finished. err: <nil> Duration: 6.070716363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..................................X.............X] duration: 1m0.002769458s trace: https://jaeger.dev.storj.io/trace/5393c5ff02b0ec06 FAIL (2025-12-27 05:32:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:31:59.143394277 (518h2m59.014477323s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144019849 (518h2m59.015102884s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144135338 (518h2m59.015218373s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.144743948 (518h2m59.015826983s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145691442 (518h2m59.016774487s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.145873368 (518h2m59.016956413s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146237642 (518h2m59.017320687s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146415481 (518h2m59.017498526s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.146884623 (518h2m59.017967669s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147158154 (518h2m59.0182412s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.147450291 (518h2m59.018533326s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.149361740 (518h2m59.020444775s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.153079940 (518h2m59.024162985s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.155357137 (518h2m59.026440192s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:31:59.158663463 (518h2m59.029746518s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Uploading piece... Dec 27 05:32:00.632852069 (518h3m0.503935114s) [upload=454501]: Next returned nil segment Dec 27 05:32:00.632864854 (518h3m0.503947899s) [upload=454501]: Waiting for error group managing segments... Dec 27 05:32:01.021877194 (518h3m0.89296023s) [noise=true , address=145.79.227.48:4321 , piece_id=URZ3ZTTUA4YE7TQFQDJWQ5K7YTX7TALMDUH3X3NRGBC64JO6L52Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.049867883 (518h3m0.920950929s) [noise=true , address=145.79.227.43:4171 , piece_id=CHHKPV6CT3R3ZQ3DDOIODXAT7D5SJHO53XM42CQJB3RH5HWK5NRA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.070540268 (518h3m0.941623303s) [noise=true , address=145.79.227.52:4454 , piece_id=RGVQSZXQ5FR3N2763ZJAVRC6HSAKIOYIQK6DRFZZ7GB4KRZQPKPQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.198338870 (518h3m1.069421915s) [noise=true , address=77.90.31.18:4832 , piece_id=BGZOENN7EEI32E4WF4BYRHNFLN6CKERKFILT7EGKFEYDAR67OYDA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.209342683 (518h3m1.080425728s) [noise=true , address=77.90.31.13:4633 , piece_id=JMGFQPVXPNNOCMCYGJROZFIR372IFHCIESJ42VUIXBVGOSDEMRZQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.258548691 (518h3m1.129631726s) [noise=true , address=45.91.116.10:4099 , piece_id=BBY4XSGWFVTVN7NS3CJ2LSDYBUQHMOERZSZY5RPXWXHP5QKOUETQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.300703391 (518h3m1.171786436s) [noise=true , address=77.90.31.16:4767 , piece_id=MY6HTETDDBYO7UPFESWNBDR5QBCORXONCTCKGCWL55AKPUOMZCAQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.311126078 (518h3m1.182209123s) [noise=true , address=45.91.116.8:4066 , piece_id=HOHCAEGOZIK6U4AZFWZTXUBFJYCCVB27YGB4G6PJSZ26A3UOSOSA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.321570256 (518h3m1.192653311s) [noise=true , address=77.90.31.19:4849 , piece_id=7A2AHVSX4IQ4YUSARAWVHKKO75SSI2KHVUQWXCBDIL5ALWPJFZ3Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.341653468 (518h3m1.212736513s) [noise=true , address=77.90.31.12:4596 , piece_id=42Z25EAXU364DEPFZKRYXIAT2SDTJZLAGV2JUMEZVC5GAAZJ2LQA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.581007482 (518h3m1.452090527s) [noise=true , address=45.91.116.13:4173 , piece_id=UWZU4LAPA2ULBLOSNIKMDQALHDYINA2LJQMSH4FMGUKVIXHA3KPA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.925980314 (518h3m1.797063359s) [noise=true , address=145.79.227.44:4219 , piece_id=5JRPI25K74UXVC6DOBNEHE3QLLFHUIZV7KJSK54IJAC5ZFEJYE7Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:01.935868202 (518h3m1.806951247s) [noise=true , address=145.79.227.34:4928 , piece_id=R5YTDJAL62CCI3CGGIEJ6YFVERC2HIMHOLLAM23RRMCSU7TTJ44Q, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163636665 (518h3m2.03471971s) [noise=true , address=77.90.31.17:4802 , piece_id=HKVYZWYJANI4XTTE25YZAHAWOWOVMQORTTOE3QPWKEPLWLBQJRXQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: <nil> Dec 27 05:32:02.163664828 (518h3m2.034747873s) [seg_pos={0 0}, upload=454501]: Segment reached optimal threshold of 14 pieces. Dec 27 05:32:02.163934743 (518h3m2.035017788s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56056->64.49.13.169:4715: use of closed network connection Dec 27 05:32:02.163945372 (518h3m2.035028418s) [noise=true , address=64.49.13.169:4715 , piece_id=YHNVJIQI2MUZAYXTXFOHZEJ5WM6Q4ADDEZQV5SPT4WJ5WC5VXUOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164336457 (518h3m2.035419502s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164346046 (518h3m2.035429091s) [noise=true , address=145.79.227.40:4064 , piece_id=HEQUY3L6KLZKL6H4Z6JYSMW26IVA25JHDGFZ4M2LLDTLXFXMPHSQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164572777 (518h3m2.035655832s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:02.164581594 (518h3m2.035664639s) [noise=true , address=145.79.227.30:4778 , piece_id=C3WUNQPOKC44ZYA242JQODDXP2WDWE45GK77VJ2LCMUODVPZAPDQ, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164777778 (518h3m2.035860823s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52076->77.90.31.10:4549: use of closed network connection Dec 27 05:32:02.164787466 (518h3m2.035870511s) [noise=true , address=77.90.31.10:4549 , piece_id=WICTN46ZKTDHLPVDFPCJZWFIIO4XCMGNEQJJMSQD53QTBTODQYOA, seg_pos={0 0}, upload=454501]: Long tail cancellation due to optimal threshold Dec 27 05:32:02.164815660 (518h3m2.035898705s) [seg_pos={0 0}, upload=454501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:32:02.164919588 (518h3m2.036002633s) [upload=454501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:02.164926110 (518h3m2.036009155s) [upload=454501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:02.463534596 (518h3m2.334617641s) [upload=454501]: Done uploading segments... Dec 27 05:32:02.463558371 (518h3m2.334641417s) [upload=454501]: Upload finished. err: <nil> Duration: 1m0.002792951s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.537671304s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:32:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537703356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...X.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.X] duration: 30.002463578s trace: https://jaeger.dev.storj.io/trace/2e2ac57bb762310 FAIL (2025-12-27 05:32:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: ZJLRO7VJZN3UANXPNFADTPCU2GN6L746CE3NTATHH33KF3J2HVRQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 6R7QOMQLWBBNR5FTZV5U242EYRRR4VR34ANJ5TYVZH4VOHL6WSHQ) context deadline exceeded ======LOGS====== Dec 27 05:32:01.185993362 (518h3m1.057076398s) [upload=454503]: Starting upload Dec 27 05:32:01.186051783 (518h3m1.057134818s) [upload=454503]: Done starting upload Dec 27 05:32:01.186233239 (518h3m1.057316274s) [upload=454503]: Uploading segments... Dec 27 05:32:01.186263216 (518h3m1.057346251s) [upload=454503]: Got next segment. Inline: false Dec 27 05:32:01.186295547 (518h3m1.057378592s) [upload=454503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:32:01.186299264 (518h3m1.057382309s) [upload=454503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:32:01.434283031 (518h3m1.305366086s) [seg_pos={0 0}, upload=454503]: Begin upload segment... Dec 27 05:32:01.434342294 (518h3m1.305425339s) [seg_pos={0 0}, upload=454503]: Done begin upload segment. Dec 27 05:32:01.434379726 (518h3m1.305462771s) [noise=true , address=45.91.116.7:4011 , piece_id=7NTRUPQSHZBB7SNSVM3KND5JPSZDYUYBU44D7SXDNNO4ER6XPUNQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.435624707 (518h3m1.306707782s) [noise=true , address=145.79.227.9:4366 , piece_id=C22XTURJLVKO7XWTDCWZYV2COIWR3Y3EBOGPZ67J6IV46Z6D5TKA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.436425772 (518h3m1.307508817s) [noise=true , address=45.91.116.13:4154 , piece_id=ZLSCTC2EMVNTTY3457LLJ2V7SGHSO6OIWN4XRCO7N3YHH4DFWHSA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.436929260 (518h3m1.308012305s) [noise=true , address=145.79.227.38:4979 , piece_id=DNZIH2DETCQLY7Q346RGBKZHKEZC7KVRUAYLQT3MNY5NRHU54WSQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.437494556 (518h3m1.308577611s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJLRO7VJZN3UANXPNFADTPCU2GN6L746CE3NTATHH33KF3J2HVRQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.439432878 (518h3m1.310515953s) [noise=true , address=145.79.227.28:4726 , piece_id=PT5IUNEPAUDKTULK4HN23AT4ZLROYYWH6ZUVXCBCKNRSZPTDUGNA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.445627834 (518h3m1.316710879s) [noise=true , address=145.79.227.51:4410 , piece_id=7J3NAEL44CC5WQNDBNVOCPTGTJLK6YK2MJYPJ2RB2EYXW6FHRR2Q, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.452350936 (518h3m1.323433981s) [noise=true , address=77.90.31.17:4770 , piece_id=5VBYLOFNVHX7GKLG7VQJ6UCFQHBOY3ONFOQODN57TPE34RLRK46Q, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.455498238 (518h3m1.326581293s) [noise=true , address=145.79.227.45:4229 , piece_id=KLUIGZWANQEXK6UYB5X3HUC2743KMYD7I42CVF6G37IGRRA6I56A, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.460578172 (518h3m1.331661227s) [noise=true , address=145.79.227.47:4267 , piece_id=M53IWGBGDHNUASN5UCJVHCKOV2SNPOCTHYIQR4IYL7IHB6R22AKA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.462177688 (518h3m1.333260733s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPGN6QD7GK5RBFJRHLMEL52VS677WM7Q7AWLUYNNKMF54TNCROZA, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.466184448 (518h3m1.337267493s) [noise=true , address=45.91.116.10:4082 , piece_id=6R7QOMQLWBBNR5FTZV5U242EYRRR4VR34ANJ5TYVZH4VOHL6WSHQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.470217087 (518h3m1.341300142s) [noise=true , address=145.79.227.48:4301 , piece_id=DLCSM2GDNUZAIKRZMVHJZ7HGVGKGGX5F2D6TW2T4KCQE24MKUERQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.471269841 (518h3m1.342352876s) [noise=true , address=45.91.116.4:4939 , piece_id=S7DWNUJXPZPBPNWHY7GNAIKKT3WZJDOU3262NG7QE4NNH5L5BG6A, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.473323872 (518h3m1.344406917s) [noise=true , address=145.79.227.46:4602 , piece_id=67MSYE2PNVLEJVKVKHUO7MOOJZZFDSB3HAEY3GH3OSFPEYTF26GQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:01.476104126 (518h3m1.347187162s) [noise=true , address=145.79.227.42:4122 , piece_id=PWUEMAWAKH2B2AN5K3I3MGHHXXQGRPSWP2LK2N2A2KYDF5HVGDXQ, seg_pos={0 0}, upload=454503]: Uploading piece... Dec 27 05:32:03.174272812 (518h3m3.045355857s) [upload=454503]: Next returned nil segment Dec 27 05:32:03.174289494 (518h3m3.045372539s) [upload=454503]: Waiting for error group managing segments... Dec 27 05:32:03.461700297 (518h3m3.332783342s) [noise=true , address=145.79.227.48:4301 , piece_id=DLCSM2GDNUZAIKRZMVHJZ7HGVGKGGX5F2D6TW2T4KCQE24MKUERQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.466944863 (518h3m3.338027909s) [noise=true , address=145.79.227.45:4229 , piece_id=KLUIGZWANQEXK6UYB5X3HUC2743KMYD7I42CVF6G37IGRRA6I56A, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.468292259 (518h3m3.339375294s) [noise=true , address=145.79.227.47:4267 , piece_id=M53IWGBGDHNUASN5UCJVHCKOV2SNPOCTHYIQR4IYL7IHB6R22AKA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.470285815 (518h3m3.34136886s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPGN6QD7GK5RBFJRHLMEL52VS677WM7Q7AWLUYNNKMF54TNCROZA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.473713602 (518h3m3.344796637s) [noise=true , address=145.79.227.42:4122 , piece_id=PWUEMAWAKH2B2AN5K3I3MGHHXXQGRPSWP2LK2N2A2KYDF5HVGDXQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.475040560 (518h3m3.346123605s) [noise=true , address=145.79.227.38:4979 , piece_id=DNZIH2DETCQLY7Q346RGBKZHKEZC7KVRUAYLQT3MNY5NRHU54WSQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.485090185 (518h3m3.35617323s) [noise=true , address=145.79.227.51:4410 , piece_id=7J3NAEL44CC5WQNDBNVOCPTGTJLK6YK2MJYPJ2RB2EYXW6FHRR2Q, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.485434701 (518h3m3.356517746s) [noise=true , address=145.79.227.9:4366 , piece_id=C22XTURJLVKO7XWTDCWZYV2COIWR3Y3EBOGPZ67J6IV46Z6D5TKA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.487168353 (518h3m3.358251398s) [noise=true , address=145.79.227.46:4602 , piece_id=67MSYE2PNVLEJVKVKHUO7MOOJZZFDSB3HAEY3GH3OSFPEYTF26GQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.497161140 (518h3m3.368244186s) [noise=true , address=145.79.227.28:4726 , piece_id=PT5IUNEPAUDKTULK4HN23AT4ZLROYYWH6ZUVXCBCKNRSZPTDUGNA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.698386150 (518h3m3.569469195s) [noise=true , address=45.91.116.10:4082 , piece_id=6R7QOMQLWBBNR5FTZV5U242EYRRR4VR34ANJ5TYVZH4VOHL6WSHQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.780186553 (518h3m3.651269598s) [noise=true , address=45.91.116.4:4939 , piece_id=S7DWNUJXPZPBPNWHY7GNAIKKT3WZJDOU3262NG7QE4NNH5L5BG6A, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.852027192 (518h3m3.723110237s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJLRO7VJZN3UANXPNFADTPCU2GN6L746CE3NTATHH33KF3J2HVRQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: <nil> Dec 27 05:32:03.852055425 (518h3m3.723138461s) [seg_pos={0 0}, upload=454503]: Segment reached optimal threshold of 13 pieces. Dec 27 05:32:03.852303117 (518h3m3.723386172s) [noise=true , address=45.91.116.7:4011 , piece_id=7NTRUPQSHZBB7SNSVM3KND5JPSZDYUYBU44D7SXDNNO4ER6XPUNQ, seg_pos={0 0}, upload=454503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:03.852312875 (518h3m3.72339592s) [noise=true , address=45.91.116.7:4011 , piece_id=7NTRUPQSHZBB7SNSVM3KND5JPSZDYUYBU44D7SXDNNO4ER6XPUNQ, seg_pos={0 0}, upload=454503]: Long tail cancellation due to optimal threshold Dec 27 05:32:03.852612236 (518h3m3.723695291s) [noise=true , address=45.91.116.13:4154 , piece_id=ZLSCTC2EMVNTTY3457LLJ2V7SGHSO6OIWN4XRCO7N3YHH4DFWHSA, seg_pos={0 0}, upload=454503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:03.852621022 (518h3m3.723704067s) [noise=true , address=45.91.116.13:4154 , piece_id=ZLSCTC2EMVNTTY3457LLJ2V7SGHSO6OIWN4XRCO7N3YHH4DFWHSA, seg_pos={0 0}, upload=454503]: Long tail cancellation due to optimal threshold Dec 27 05:32:03.852793421 (518h3m3.723876456s) [noise=true , address=77.90.31.17:4770 , piece_id=5VBYLOFNVHX7GKLG7VQJ6UCFQHBOY3ONFOQODN57TPE34RLRK46Q, seg_pos={0 0}, upload=454503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 05:32:03.852802698 (518h3m3.723885733s) [noise=true , address=77.90.31.17:4770 , piece_id=5VBYLOFNVHX7GKLG7VQJ6UCFQHBOY3ONFOQODN57TPE34RLRK46Q, seg_pos={0 0}, upload=454503]: Long tail cancellation due to optimal threshold Dec 27 05:32:03.852827646 (518h3m3.723910691s) [seg_pos={0 0}, upload=454503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:32:03.852924460 (518h3m3.724007495s) [upload=454503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:32:03.852932045 (518h3m3.72401509s) [upload=454503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:32:04.152690630 (518h3m4.023773675s) [upload=454503]: Done uploading segments... Dec 27 05:32:04.152715217 (518h3m4.023798272s) [upload=454503]: Upload finished. err: <nil> Duration: 30.002476382s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 4.79431569s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:32:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.794345326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..................................X.............] duration: 39.360351105s trace: https://jaeger.dev.storj.io/trace/63f68f1606b74937 FAIL (2025-12-27 05:31:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA) context deadline exceeded; piecestore: (Node ID: 168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx, Piece ID: JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ) context deadline exceeded ======LOGS====== Dec 27 05:20:06.192886534 (517h51m6.063969569s) [noise=true , address=145.79.227.2:4136 , piece_id=TEYF4XZLEEWODAH76AW3UHKUPTB4ULIC4JQTTWX77TKTQVBH7Y5Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.193332814 (517h51m6.064415859s) [noise=true , address=193.149.164.143:4699 , piece_id=BKDTTBGO7KGZBNKEILXLO4YCXC3JJHGDFNNC4IT2SL4JQTZZ4EAA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.193645419 (517h51m6.064728465s) [noise=true , address=145.79.227.44:4225 , piece_id=LVV7XRDDMZ2OKNNDM6V46C2W3K2F5KYAQAC66BWYPZFEY42OPUHQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194011196 (517h51m6.065094242s) [noise=true , address=145.79.227.34:4938 , piece_id=BSL7K2IT3O7KF6B7HEQ3TKGYYC6K42RWYXVFH6BYXZOHT2TVCNQA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194331446 (517h51m6.065414481s) [noise=true , address=77.90.31.10:4536 , piece_id=LVBZWIMVH2OMOLHSEPXXSQCHNTYMRXYRN7KS43DKTMJ2OWITC45A, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194790350 (517h51m6.065873395s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195112773 (517h51m6.066195818s) [noise=true , address=77.90.31.17:4804 , piece_id=JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195470736 (517h51m6.066553771s) [noise=true , address=145.79.227.2:4137 , piece_id=FZUUJKMD7F4SEZ6M6UIL3S7DGHKEJKWE45ZDVEFDWPE47S765FAQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195772339 (517h51m6.066855384s) [noise=true , address=145.79.227.30:4766 , piece_id=HOC57KMPUPKIT35ILDWWNN5SHSBX2EKN5C6ZLHY2HWBKFZ2QXDBQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.196239610 (517h51m6.067322645s) [noise=true , address=77.90.31.19:4848 , piece_id=EZP26DQ7FLJYKTEGV5D74BJ7WKSB3HV2WUF3YCLQDUW6HQIFOCTA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.197528574 (517h51m6.068611609s) [noise=true , address=77.90.31.14:4673 , piece_id=PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.198047893 (517h51m6.069130928s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.198645551 (517h51m6.069728596s) [noise=true , address=145.79.227.15:4903 , piece_id=HITMQCCZLOT4JOZZURN25CFTNI5NWOPZY4P46MD6RIUEIZXMFUPA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199105326 (517h51m6.070188361s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199740606 (517h51m6.070823641s) [noise=true , address=145.79.227.10:4436 , piece_id=22RZSZO265VEXKQVTAUQYRT223VKQY7N4DNAP5NFAOK5DH44MQAQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199909958 (517h51m6.070993003s) [noise=true , address=77.90.31.13:4647 , piece_id=GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:07.569793709 (517h51m7.440876764s) [upload=454334]: Next returned nil segment Dec 27 05:20:07.569833555 (517h51m7.44091662s) [upload=454334]: Waiting for error group managing segments... Dec 27 05:20:07.922714305 (517h51m7.79379735s) [noise=true , address=193.149.164.143:4699 , piece_id=BKDTTBGO7KGZBNKEILXLO4YCXC3JJHGDFNNC4IT2SL4JQTZZ4EAA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:07.943656403 (517h51m7.814739448s) [noise=true , address=64.49.13.140:4507 , piece_id=RAIYSJOLWFJK76ZFXIKT47AUSHNNJPTU7ACE2VVPGPNWDOUJG4VQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.060758757 (517h51m7.931841802s) [noise=true , address=145.79.227.10:4436 , piece_id=22RZSZO265VEXKQVTAUQYRT223VKQY7N4DNAP5NFAOK5DH44MQAQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.064833787 (517h51m7.935916842s) [noise=true , address=145.79.227.15:4903 , piece_id=HITMQCCZLOT4JOZZURN25CFTNI5NWOPZY4P46MD6RIUEIZXMFUPA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.067874817 (517h51m7.938957852s) [noise=true , address=145.79.227.2:4137 , piece_id=FZUUJKMD7F4SEZ6M6UIL3S7DGHKEJKWE45ZDVEFDWPE47S765FAQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.101063785 (517h51m7.97214682s) [noise=true , address=145.79.227.2:4136 , piece_id=TEYF4XZLEEWODAH76AW3UHKUPTB4ULIC4JQTTWX77TKTQVBH7Y5Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.281421209 (517h51m8.152504254s) [noise=true , address=77.90.31.13:4647 , piece_id=GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.292502911 (517h51m8.163585956s) [noise=true , address=77.90.31.19:4848 , piece_id=EZP26DQ7FLJYKTEGV5D74BJ7WKSB3HV2WUF3YCLQDUW6HQIFOCTA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.347820305 (517h51m8.21890335s) [noise=true , address=77.90.31.14:4673 , piece_id=PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.369437187 (517h51m8.240520222s) [noise=true , address=77.90.31.10:4536 , piece_id=LVBZWIMVH2OMOLHSEPXXSQCHNTYMRXYRN7KS43DKTMJ2OWITC45A, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.623303356 (517h51m8.494386401s) [noise=true , address=77.90.31.17:4804 , piece_id=JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.970073139 (517h51m8.841156184s) [noise=true , address=145.79.227.44:4225 , piece_id=LVV7XRDDMZ2OKNNDM6V46C2W3K2F5KYAQAC66BWYPZFEY42OPUHQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.004503902 (517h51m8.875586947s) [noise=true , address=145.79.227.30:4766 , piece_id=HOC57KMPUPKIT35ILDWWNN5SHSBX2EKN5C6ZLHY2HWBKFZ2QXDBQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.064493524 (517h51m8.935576569s) [noise=true , address=145.79.227.34:4938 , piece_id=BSL7K2IT3O7KF6B7HEQ3TKGYYC6K42RWYXVFH6BYXZOHT2TVCNQA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.064523371 (517h51m8.935606416s) [seg_pos={0 0}, upload=454334]: Segment reached optimal threshold of 14 pieces. Dec 27 05:20:09.064788426 (517h51m8.935871471s) [noise=true , address=64.49.13.169:4707 , piece_id=AIIKCSLONZRP6G2VLKZVS5XXA33RZKZEEKU5JKK34OQE55YZARCQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection Dec 27 05:20:09.064801250 (517h51m8.935884295s) [noise=true , address=64.49.13.169:4707 , piece_id=AIIKCSLONZRP6G2VLKZVS5XXA33RZKZEEKU5JKK34OQE55YZARCQ, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.064881923 (517h51m8.935964958s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.064899587 (517h51m8.935982632s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065087855 (517h51m8.9361709s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.065098886 (517h51m8.936181921s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065174411 (517h51m8.936257476s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.065186092 (517h51m8.936269138s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065214106 (517h51m8.936297141s) [seg_pos={0 0}, upload=454334]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:20:09.065313125 (517h51m8.93639617s) [upload=454334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:20:09.065321080 (517h51m8.936404125s) [upload=454334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:20:09.374729791 (517h51m9.245812846s) [upload=454334]: Done uploading segments... Dec 27 05:20:09.374755670 (517h51m9.245838706s) [upload=454334]: Upload finished. err: <nil> Duration: 39.36036996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X...X.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX.] duration: 6.006927154s trace: https://jaeger.dev.storj.io/trace/5d4ebc942f16d347 FAIL (2025-12-27 05:31:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 05:30:49.582346976 (518h1m49.453430011s) [upload=454484]: Starting upload Dec 27 05:30:49.582424844 (518h1m49.453507879s) [upload=454484]: Done starting upload Dec 27 05:30:49.582605088 (518h1m49.453688123s) [upload=454484]: Uploading segments... Dec 27 05:30:49.582655855 (518h1m49.45373889s) [upload=454484]: Got next segment. Inline: false Dec 27 05:30:49.582759572 (518h1m49.453842607s) [upload=454484]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:30:49.582763860 (518h1m49.453846895s) [upload=454484]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:30:49.828400117 (518h1m49.699483162s) [seg_pos={0 0}, upload=454484]: Begin upload segment... Dec 27 05:30:49.828506770 (518h1m49.699589805s) [seg_pos={0 0}, upload=454484]: Done begin upload segment. Dec 27 05:30:49.828549482 (518h1m49.699632517s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRXD4B77XOP3WV6EP56TG2XMPEKR4JSVJ6SQRONN4K3JI65B6VA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.829454896 (518h1m49.700537961s) [noise=true , address=145.79.227.13:4804 , piece_id=DIBNGHTX3LNRIKFBP76FUB6RBAJ7EAQCJNSQTJJ5NU5JDZT67GMQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.830098641 (518h1m49.701181676s) [noise=true , address=145.79.227.20:4439 , piece_id=TOUKQJ3K2QDXOI3FG36DN3VT37QIG7IIBZ3ZNHTGBVXDTVZ7A3OQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.830525353 (518h1m49.701608398s) [noise=true , address=77.90.31.10:4518 , piece_id=53GOOANXT5ZMZG4VVNVK5NMJMWACBMKKCWBSNTNSLLZMN2PW5VHQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.830941466 (518h1m49.702024501s) [noise=true , address=45.91.116.13:4154 , piece_id=K6SZ5ZXNS7KTRFRI2TG5PVO6QD6P27KKXLNJMDLIPKUCHGT2JA4Q, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.831479611 (518h1m49.702562646s) [noise=true , address=77.90.31.14:4663 , piece_id=AF6RJ63KHI7GYS3PHDWP3OAXSKNC3PX3PPLOZFRTLXYBCYLVWRLQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.831905231 (518h1m49.702988276s) [noise=true , address=145.79.227.38:4979 , piece_id=PCU33KB6NLDQO6QFCJTPHM44743YUYBFXGRPKV3VGSBPA7VNCQOQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.832310724 (518h1m49.703393799s) [noise=true , address=145.79.227.41:4086 , piece_id=6QTTVXJH6CUO3NPMQTH4GFAENHDMEH3TUUEFFRNSZKGVECTLUCBA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.832746593 (518h1m49.703829638s) [noise=true , address=145.79.227.42:4123 , piece_id=DOCAUH5L2PSFLCHWVFK4PKR675CKET7NSGMUE75EVM45VONOVLFA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.833820368 (518h1m49.704903403s) [noise=true , address=145.79.227.45:4229 , piece_id=XLXRBOFLSPFF6WIKMAV4MR2YGX5S7NO3K5HZUEIWBUEONGN4JCOA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.834192357 (518h1m49.705275402s) [noise=true , address=145.79.227.17:4970 , piece_id=EHPIV4FI26NKVOPF2L4UQLWTQ2MNJ2ZSUDDPZV6W6ANGF57SGG5Q, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.834627505 (518h1m49.70571055s) [noise=true , address=45.91.116.12:4228 , piece_id=A3KLY5Q664P5VBI3T66JXSJO53QVLNAAD4Q6U34DJAE7GSRZAAOA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.836295431 (518h1m49.707378466s) [noise=true , address=145.79.227.39:4014 , piece_id=Y2BOZUTHIKJJ2LHPVGZ6T4SYFM3E3GZN23SPYKPJLTVZSD3EJ5JQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.838117381 (518h1m49.709200416s) [noise=true , address=145.79.227.50:4373 , piece_id=6SWNQZXPL3BYYGAYD2PXVZNDY6T2Q3NYPYPLKXPKM5C5J2RQOYRQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.839608680 (518h1m49.710691725s) [noise=true , address=145.79.227.51:4411 , piece_id=2HAJCGC75Q734HWHN2YQMS43GERYSIY5XEQOLT3GDDTP4RHA7SJA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.841113416 (518h1m49.712196451s) [noise=true , address=145.79.227.31:4798 , piece_id=4IKPJD2WHCX4IHIGXIICT3M2UGKCPAAYKMJZG2H55UPSQSTGJ4MA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:51.457047772 (518h1m51.328130807s) [upload=454484]: Next returned nil segment Dec 27 05:30:51.457125039 (518h1m51.328208084s) [upload=454484]: Waiting for error group managing segments... Dec 27 05:30:51.831590661 (518h1m51.702673696s) [noise=true , address=145.79.227.13:4804 , piece_id=DIBNGHTX3LNRIKFBP76FUB6RBAJ7EAQCJNSQTJJ5NU5JDZT67GMQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.834894292 (518h1m51.705977337s) [noise=true , address=145.79.227.20:4439 , piece_id=TOUKQJ3K2QDXOI3FG36DN3VT37QIG7IIBZ3ZNHTGBVXDTVZ7A3OQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.842026261 (518h1m51.713109296s) [noise=true , address=145.79.227.51:4411 , piece_id=2HAJCGC75Q734HWHN2YQMS43GERYSIY5XEQOLT3GDDTP4RHA7SJA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.843265021 (518h1m51.714348056s) [noise=true , address=145.79.227.38:4979 , piece_id=PCU33KB6NLDQO6QFCJTPHM44743YUYBFXGRPKV3VGSBPA7VNCQOQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.845932480 (518h1m51.717015525s) [noise=true , address=145.79.227.39:4014 , piece_id=Y2BOZUTHIKJJ2LHPVGZ6T4SYFM3E3GZN23SPYKPJLTVZSD3EJ5JQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.848086893 (518h1m51.719169928s) [noise=true , address=145.79.227.41:4086 , piece_id=6QTTVXJH6CUO3NPMQTH4GFAENHDMEH3TUUEFFRNSZKGVECTLUCBA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.856186074 (518h1m51.727269119s) [noise=true , address=145.79.227.50:4373 , piece_id=6SWNQZXPL3BYYGAYD2PXVZNDY6T2Q3NYPYPLKXPKM5C5J2RQOYRQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.863511013 (518h1m51.734594048s) [noise=true , address=145.79.227.31:4798 , piece_id=4IKPJD2WHCX4IHIGXIICT3M2UGKCPAAYKMJZG2H55UPSQSTGJ4MA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.866888113 (518h1m51.737971159s) [noise=true , address=145.79.227.42:4123 , piece_id=DOCAUH5L2PSFLCHWVFK4PKR675CKET7NSGMUE75EVM45VONOVLFA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.869760213 (518h1m51.740843248s) [noise=true , address=145.79.227.17:4970 , piece_id=EHPIV4FI26NKVOPF2L4UQLWTQ2MNJ2ZSUDDPZV6W6ANGF57SGG5Q, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.870279822 (518h1m51.741362857s) [noise=true , address=145.79.227.45:4229 , piece_id=XLXRBOFLSPFF6WIKMAV4MR2YGX5S7NO3K5HZUEIWBUEONGN4JCOA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:52.073355055 (518h1m51.9444381s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRXD4B77XOP3WV6EP56TG2XMPEKR4JSVJ6SQRONN4K3JI65B6VA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:52.099450084 (518h1m51.970533119s) [noise=true , address=45.91.116.12:4228 , piece_id=A3KLY5Q664P5VBI3T66JXSJO53QVLNAAD4Q6U34DJAE7GSRZAAOA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:52.099470493 (518h1m51.970553528s) [seg_pos={0 0}, upload=454484]: Segment reached optimal threshold of 13 pieces. Dec 27 05:30:52.099700020 (518h1m51.970783065s) [noise=true , address=77.90.31.14:4663 , piece_id=AF6RJ63KHI7GYS3PHDWP3OAXSKNC3PX3PPLOZFRTLXYBCYLVWRLQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:59180->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:59180->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59180->77.90.31.14:4663: use of closed network connection Dec 27 05:30:52.099707414 (518h1m51.970790449s) [noise=true , address=77.90.31.14:4663 , piece_id=AF6RJ63KHI7GYS3PHDWP3OAXSKNC3PX3PPLOZFRTLXYBCYLVWRLQ, seg_pos={0 0}, upload=454484]: Long tail cancellation due to optimal threshold Dec 27 05:30:52.099878100 (518h1m51.970961315s) [noise=true , address=77.90.31.10:4518 , piece_id=53GOOANXT5ZMZG4VVNVK5NMJMWACBMKKCWBSNTNSLLZMN2PW5VHQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 05:30:52.099896143 (518h1m51.970979178s) [noise=true , address=77.90.31.10:4518 , piece_id=53GOOANXT5ZMZG4VVNVK5NMJMWACBMKKCWBSNTNSLLZMN2PW5VHQ, seg_pos={0 0}, upload=454484]: Long tail cancellation due to optimal threshold Dec 27 05:30:52.099924087 (518h1m51.971007122s) [noise=true , address=45.91.116.13:4154 , piece_id=K6SZ5ZXNS7KTRFRI2TG5PVO6QD6P27KKXLNJMDLIPKUCHGT2JA4Q, seg_pos={0 0}, upload=454484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 05:30:52.099931401 (518h1m51.971014436s) [noise=true , address=45.91.116.13:4154 , piece_id=K6SZ5ZXNS7KTRFRI2TG5PVO6QD6P27KKXLNJMDLIPKUCHGT2JA4Q, seg_pos={0 0}, upload=454484]: Long tail cancellation due to optimal threshold Dec 27 05:30:52.100081085 (518h1m51.97116413s) [seg_pos={0 0}, upload=454484]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:30:52.100199652 (518h1m51.971282687s) [upload=454484]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:30:52.100205242 (518h1m51.971288287s) [upload=454484]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:30:52.389585966 (518h1m52.260669011s) [upload=454484]: Done uploading segments... Dec 27 05:30:52.389640721 (518h1m52.260723766s) [upload=454484]: Upload finished. err: <nil> Duration: 6.006959275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX...X.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..XX] duration: 30.003256418s trace: https://jaeger.dev.storj.io/trace/5d4ebc942f16d347 FAIL (2025-12-27 05:31:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 05:30:49.582346976 (518h1m49.453430011s) [upload=454484]: Starting upload Dec 27 05:30:49.582424844 (518h1m49.453507879s) [upload=454484]: Done starting upload Dec 27 05:30:49.582605088 (518h1m49.453688123s) [upload=454484]: Uploading segments... Dec 27 05:30:49.582655855 (518h1m49.45373889s) [upload=454484]: Got next segment. Inline: false Dec 27 05:30:49.582759572 (518h1m49.453842607s) [upload=454484]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:30:49.582763860 (518h1m49.453846895s) [upload=454484]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:30:49.828400117 (518h1m49.699483162s) [seg_pos={0 0}, upload=454484]: Begin upload segment... Dec 27 05:30:49.828506770 (518h1m49.699589805s) [seg_pos={0 0}, upload=454484]: Done begin upload segment. Dec 27 05:30:49.828549482 (518h1m49.699632517s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRXD4B77XOP3WV6EP56TG2XMPEKR4JSVJ6SQRONN4K3JI65B6VA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.829454896 (518h1m49.700537961s) [noise=true , address=145.79.227.13:4804 , piece_id=DIBNGHTX3LNRIKFBP76FUB6RBAJ7EAQCJNSQTJJ5NU5JDZT67GMQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.830098641 (518h1m49.701181676s) [noise=true , address=145.79.227.20:4439 , piece_id=TOUKQJ3K2QDXOI3FG36DN3VT37QIG7IIBZ3ZNHTGBVXDTVZ7A3OQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.830525353 (518h1m49.701608398s) [noise=true , address=77.90.31.10:4518 , piece_id=53GOOANXT5ZMZG4VVNVK5NMJMWACBMKKCWBSNTNSLLZMN2PW5VHQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.830941466 (518h1m49.702024501s) [noise=true , address=45.91.116.13:4154 , piece_id=K6SZ5ZXNS7KTRFRI2TG5PVO6QD6P27KKXLNJMDLIPKUCHGT2JA4Q, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.831479611 (518h1m49.702562646s) [noise=true , address=77.90.31.14:4663 , piece_id=AF6RJ63KHI7GYS3PHDWP3OAXSKNC3PX3PPLOZFRTLXYBCYLVWRLQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.831905231 (518h1m49.702988276s) [noise=true , address=145.79.227.38:4979 , piece_id=PCU33KB6NLDQO6QFCJTPHM44743YUYBFXGRPKV3VGSBPA7VNCQOQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.832310724 (518h1m49.703393799s) [noise=true , address=145.79.227.41:4086 , piece_id=6QTTVXJH6CUO3NPMQTH4GFAENHDMEH3TUUEFFRNSZKGVECTLUCBA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.832746593 (518h1m49.703829638s) [noise=true , address=145.79.227.42:4123 , piece_id=DOCAUH5L2PSFLCHWVFK4PKR675CKET7NSGMUE75EVM45VONOVLFA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.833820368 (518h1m49.704903403s) [noise=true , address=145.79.227.45:4229 , piece_id=XLXRBOFLSPFF6WIKMAV4MR2YGX5S7NO3K5HZUEIWBUEONGN4JCOA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.834192357 (518h1m49.705275402s) [noise=true , address=145.79.227.17:4970 , piece_id=EHPIV4FI26NKVOPF2L4UQLWTQ2MNJ2ZSUDDPZV6W6ANGF57SGG5Q, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.834627505 (518h1m49.70571055s) [noise=true , address=45.91.116.12:4228 , piece_id=A3KLY5Q664P5VBI3T66JXSJO53QVLNAAD4Q6U34DJAE7GSRZAAOA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.836295431 (518h1m49.707378466s) [noise=true , address=145.79.227.39:4014 , piece_id=Y2BOZUTHIKJJ2LHPVGZ6T4SYFM3E3GZN23SPYKPJLTVZSD3EJ5JQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.838117381 (518h1m49.709200416s) [noise=true , address=145.79.227.50:4373 , piece_id=6SWNQZXPL3BYYGAYD2PXVZNDY6T2Q3NYPYPLKXPKM5C5J2RQOYRQ, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.839608680 (518h1m49.710691725s) [noise=true , address=145.79.227.51:4411 , piece_id=2HAJCGC75Q734HWHN2YQMS43GERYSIY5XEQOLT3GDDTP4RHA7SJA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:49.841113416 (518h1m49.712196451s) [noise=true , address=145.79.227.31:4798 , piece_id=4IKPJD2WHCX4IHIGXIICT3M2UGKCPAAYKMJZG2H55UPSQSTGJ4MA, seg_pos={0 0}, upload=454484]: Uploading piece... Dec 27 05:30:51.457047772 (518h1m51.328130807s) [upload=454484]: Next returned nil segment Dec 27 05:30:51.457125039 (518h1m51.328208084s) [upload=454484]: Waiting for error group managing segments... Dec 27 05:30:51.831590661 (518h1m51.702673696s) [noise=true , address=145.79.227.13:4804 , piece_id=DIBNGHTX3LNRIKFBP76FUB6RBAJ7EAQCJNSQTJJ5NU5JDZT67GMQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.834894292 (518h1m51.705977337s) [noise=true , address=145.79.227.20:4439 , piece_id=TOUKQJ3K2QDXOI3FG36DN3VT37QIG7IIBZ3ZNHTGBVXDTVZ7A3OQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.842026261 (518h1m51.713109296s) [noise=true , address=145.79.227.51:4411 , piece_id=2HAJCGC75Q734HWHN2YQMS43GERYSIY5XEQOLT3GDDTP4RHA7SJA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.843265021 (518h1m51.714348056s) [noise=true , address=145.79.227.38:4979 , piece_id=PCU33KB6NLDQO6QFCJTPHM44743YUYBFXGRPKV3VGSBPA7VNCQOQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.845932480 (518h1m51.717015525s) [noise=true , address=145.79.227.39:4014 , piece_id=Y2BOZUTHIKJJ2LHPVGZ6T4SYFM3E3GZN23SPYKPJLTVZSD3EJ5JQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.848086893 (518h1m51.719169928s) [noise=true , address=145.79.227.41:4086 , piece_id=6QTTVXJH6CUO3NPMQTH4GFAENHDMEH3TUUEFFRNSZKGVECTLUCBA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.856186074 (518h1m51.727269119s) [noise=true , address=145.79.227.50:4373 , piece_id=6SWNQZXPL3BYYGAYD2PXVZNDY6T2Q3NYPYPLKXPKM5C5J2RQOYRQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.863511013 (518h1m51.734594048s) [noise=true , address=145.79.227.31:4798 , piece_id=4IKPJD2WHCX4IHIGXIICT3M2UGKCPAAYKMJZG2H55UPSQSTGJ4MA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.866888113 (518h1m51.737971159s) [noise=true , address=145.79.227.42:4123 , piece_id=DOCAUH5L2PSFLCHWVFK4PKR675CKET7NSGMUE75EVM45VONOVLFA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.869760213 (518h1m51.740843248s) [noise=true , address=145.79.227.17:4970 , piece_id=EHPIV4FI26NKVOPF2L4UQLWTQ2MNJ2ZSUDDPZV6W6ANGF57SGG5Q, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:51.870279822 (518h1m51.741362857s) [noise=true , address=145.79.227.45:4229 , piece_id=XLXRBOFLSPFF6WIKMAV4MR2YGX5S7NO3K5HZUEIWBUEONGN4JCOA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:52.073355055 (518h1m51.9444381s) [noise=true , address=45.91.116.10:4081 , piece_id=FPRXD4B77XOP3WV6EP56TG2XMPEKR4JSVJ6SQRONN4K3JI65B6VA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:52.099450084 (518h1m51.970533119s) [noise=true , address=45.91.116.12:4228 , piece_id=A3KLY5Q664P5VBI3T66JXSJO53QVLNAAD4Q6U34DJAE7GSRZAAOA, seg_pos={0 0}, upload=454484]: Done uploading piece. err: <nil> Dec 27 05:30:52.099470493 (518h1m51.970553528s) [seg_pos={0 0}, upload=454484]: Segment reached optimal threshold of 13 pieces. Dec 27 05:30:52.099700020 (518h1m51.970783065s) [noise=true , address=77.90.31.14:4663 , piece_id=AF6RJ63KHI7GYS3PHDWP3OAXSKNC3PX3PPLOZFRTLXYBCYLVWRLQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:59180->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:59180->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59180->77.90.31.14:4663: use of closed network connection Dec 27 05:30:52.099707414 (518h1m51.970790449s) [noise=true , address=77.90.31.14:4663 , piece_id=AF6RJ63KHI7GYS3PHDWP3OAXSKNC3PX3PPLOZFRTLXYBCYLVWRLQ, seg_pos={0 0}, upload=454484]: Long tail cancellation due to optimal threshold Dec 27 05:30:52.099878100 (518h1m51.970961315s) [noise=true , address=77.90.31.10:4518 , piece_id=53GOOANXT5ZMZG4VVNVK5NMJMWACBMKKCWBSNTNSLLZMN2PW5VHQ, seg_pos={0 0}, upload=454484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 05:30:52.099896143 (518h1m51.970979178s) [noise=true , address=77.90.31.10:4518 , piece_id=53GOOANXT5ZMZG4VVNVK5NMJMWACBMKKCWBSNTNSLLZMN2PW5VHQ, seg_pos={0 0}, upload=454484]: Long tail cancellation due to optimal threshold Dec 27 05:30:52.099924087 (518h1m51.971007122s) [noise=true , address=45.91.116.13:4154 , piece_id=K6SZ5ZXNS7KTRFRI2TG5PVO6QD6P27KKXLNJMDLIPKUCHGT2JA4Q, seg_pos={0 0}, upload=454484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 05:30:52.099931401 (518h1m51.971014436s) [noise=true , address=45.91.116.13:4154 , piece_id=K6SZ5ZXNS7KTRFRI2TG5PVO6QD6P27KKXLNJMDLIPKUCHGT2JA4Q, seg_pos={0 0}, upload=454484]: Long tail cancellation due to optimal threshold Dec 27 05:30:52.100081085 (518h1m51.97116413s) [seg_pos={0 0}, upload=454484]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:30:52.100199652 (518h1m51.971282687s) [upload=454484]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:30:52.100205242 (518h1m51.971288287s) [upload=454484]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:30:52.389585966 (518h1m52.260669011s) [upload=454484]: Done uploading segments... Dec 27 05:30:52.389640721 (518h1m52.260723766s) [upload=454484]: Upload finished. err: <nil> Duration: 30.003281544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..................................X............] duration: 54.9319443s trace: https://jaeger.dev.storj.io/trace/63f68f1606b74937 FAIL (2025-12-27 05:31:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA) context deadline exceeded; piecestore: (Node ID: 168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx, Piece ID: JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ) context deadline exceeded ======LOGS====== Dec 27 05:20:06.192886534 (517h51m6.063969569s) [noise=true , address=145.79.227.2:4136 , piece_id=TEYF4XZLEEWODAH76AW3UHKUPTB4ULIC4JQTTWX77TKTQVBH7Y5Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.193332814 (517h51m6.064415859s) [noise=true , address=193.149.164.143:4699 , piece_id=BKDTTBGO7KGZBNKEILXLO4YCXC3JJHGDFNNC4IT2SL4JQTZZ4EAA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.193645419 (517h51m6.064728465s) [noise=true , address=145.79.227.44:4225 , piece_id=LVV7XRDDMZ2OKNNDM6V46C2W3K2F5KYAQAC66BWYPZFEY42OPUHQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194011196 (517h51m6.065094242s) [noise=true , address=145.79.227.34:4938 , piece_id=BSL7K2IT3O7KF6B7HEQ3TKGYYC6K42RWYXVFH6BYXZOHT2TVCNQA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194331446 (517h51m6.065414481s) [noise=true , address=77.90.31.10:4536 , piece_id=LVBZWIMVH2OMOLHSEPXXSQCHNTYMRXYRN7KS43DKTMJ2OWITC45A, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194790350 (517h51m6.065873395s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195112773 (517h51m6.066195818s) [noise=true , address=77.90.31.17:4804 , piece_id=JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195470736 (517h51m6.066553771s) [noise=true , address=145.79.227.2:4137 , piece_id=FZUUJKMD7F4SEZ6M6UIL3S7DGHKEJKWE45ZDVEFDWPE47S765FAQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195772339 (517h51m6.066855384s) [noise=true , address=145.79.227.30:4766 , piece_id=HOC57KMPUPKIT35ILDWWNN5SHSBX2EKN5C6ZLHY2HWBKFZ2QXDBQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.196239610 (517h51m6.067322645s) [noise=true , address=77.90.31.19:4848 , piece_id=EZP26DQ7FLJYKTEGV5D74BJ7WKSB3HV2WUF3YCLQDUW6HQIFOCTA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.197528574 (517h51m6.068611609s) [noise=true , address=77.90.31.14:4673 , piece_id=PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.198047893 (517h51m6.069130928s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.198645551 (517h51m6.069728596s) [noise=true , address=145.79.227.15:4903 , piece_id=HITMQCCZLOT4JOZZURN25CFTNI5NWOPZY4P46MD6RIUEIZXMFUPA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199105326 (517h51m6.070188361s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199740606 (517h51m6.070823641s) [noise=true , address=145.79.227.10:4436 , piece_id=22RZSZO265VEXKQVTAUQYRT223VKQY7N4DNAP5NFAOK5DH44MQAQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199909958 (517h51m6.070993003s) [noise=true , address=77.90.31.13:4647 , piece_id=GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:07.569793709 (517h51m7.440876764s) [upload=454334]: Next returned nil segment Dec 27 05:20:07.569833555 (517h51m7.44091662s) [upload=454334]: Waiting for error group managing segments... Dec 27 05:20:07.922714305 (517h51m7.79379735s) [noise=true , address=193.149.164.143:4699 , piece_id=BKDTTBGO7KGZBNKEILXLO4YCXC3JJHGDFNNC4IT2SL4JQTZZ4EAA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:07.943656403 (517h51m7.814739448s) [noise=true , address=64.49.13.140:4507 , piece_id=RAIYSJOLWFJK76ZFXIKT47AUSHNNJPTU7ACE2VVPGPNWDOUJG4VQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.060758757 (517h51m7.931841802s) [noise=true , address=145.79.227.10:4436 , piece_id=22RZSZO265VEXKQVTAUQYRT223VKQY7N4DNAP5NFAOK5DH44MQAQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.064833787 (517h51m7.935916842s) [noise=true , address=145.79.227.15:4903 , piece_id=HITMQCCZLOT4JOZZURN25CFTNI5NWOPZY4P46MD6RIUEIZXMFUPA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.067874817 (517h51m7.938957852s) [noise=true , address=145.79.227.2:4137 , piece_id=FZUUJKMD7F4SEZ6M6UIL3S7DGHKEJKWE45ZDVEFDWPE47S765FAQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.101063785 (517h51m7.97214682s) [noise=true , address=145.79.227.2:4136 , piece_id=TEYF4XZLEEWODAH76AW3UHKUPTB4ULIC4JQTTWX77TKTQVBH7Y5Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.281421209 (517h51m8.152504254s) [noise=true , address=77.90.31.13:4647 , piece_id=GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.292502911 (517h51m8.163585956s) [noise=true , address=77.90.31.19:4848 , piece_id=EZP26DQ7FLJYKTEGV5D74BJ7WKSB3HV2WUF3YCLQDUW6HQIFOCTA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.347820305 (517h51m8.21890335s) [noise=true , address=77.90.31.14:4673 , piece_id=PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.369437187 (517h51m8.240520222s) [noise=true , address=77.90.31.10:4536 , piece_id=LVBZWIMVH2OMOLHSEPXXSQCHNTYMRXYRN7KS43DKTMJ2OWITC45A, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.623303356 (517h51m8.494386401s) [noise=true , address=77.90.31.17:4804 , piece_id=JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.970073139 (517h51m8.841156184s) [noise=true , address=145.79.227.44:4225 , piece_id=LVV7XRDDMZ2OKNNDM6V46C2W3K2F5KYAQAC66BWYPZFEY42OPUHQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.004503902 (517h51m8.875586947s) [noise=true , address=145.79.227.30:4766 , piece_id=HOC57KMPUPKIT35ILDWWNN5SHSBX2EKN5C6ZLHY2HWBKFZ2QXDBQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.064493524 (517h51m8.935576569s) [noise=true , address=145.79.227.34:4938 , piece_id=BSL7K2IT3O7KF6B7HEQ3TKGYYC6K42RWYXVFH6BYXZOHT2TVCNQA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.064523371 (517h51m8.935606416s) [seg_pos={0 0}, upload=454334]: Segment reached optimal threshold of 14 pieces. Dec 27 05:20:09.064788426 (517h51m8.935871471s) [noise=true , address=64.49.13.169:4707 , piece_id=AIIKCSLONZRP6G2VLKZVS5XXA33RZKZEEKU5JKK34OQE55YZARCQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection Dec 27 05:20:09.064801250 (517h51m8.935884295s) [noise=true , address=64.49.13.169:4707 , piece_id=AIIKCSLONZRP6G2VLKZVS5XXA33RZKZEEKU5JKK34OQE55YZARCQ, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.064881923 (517h51m8.935964958s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.064899587 (517h51m8.935982632s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065087855 (517h51m8.9361709s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.065098886 (517h51m8.936181921s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065174411 (517h51m8.936257476s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.065186092 (517h51m8.936269138s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065214106 (517h51m8.936297141s) [seg_pos={0 0}, upload=454334]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:20:09.065313125 (517h51m8.93639617s) [upload=454334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:20:09.065321080 (517h51m8.936404125s) [upload=454334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:20:09.374729791 (517h51m9.245812846s) [upload=454334]: Done uploading segments... Dec 27 05:20:09.374755670 (517h51m9.245838706s) [upload=454334]: Upload finished. err: <nil> Duration: 54.931969879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 4.106628487s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:31:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.106659667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XX...X.XX....XXXX..X.X.XX..XXX..X......XX.XXX....XXX...X..X] duration: 30.005678613s trace: https://jaeger.dev.storj.io/trace/28b83cfd9a9bc902 FAIL (2025-12-27 05:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 4WGFMS2NKFJVN3MZRZSBGBLCMZLZCVW4HBHXHIIQELTMRKDRIKLQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 66IKZOIC7VLBVFHXYIO5MBLUQTH7KR45Y53KS4XPVUU5RZQA3MVQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: OQJRZTXUWHAEIHG2RJIDXY46UTQ22NIFALGMYS2VBWNTKTPMMESA) context deadline exceeded ======LOGS====== Dec 27 05:30:01.095781459 (518h1m0.966864504s) [upload=454470]: Starting upload Dec 27 05:30:01.095844209 (518h1m0.966927244s) [upload=454470]: Done starting upload Dec 27 05:30:01.096001127 (518h1m0.967084172s) [upload=454470]: Uploading segments... Dec 27 05:30:01.096156454 (518h1m0.967239499s) [upload=454470]: Got next segment. Inline: false Dec 27 05:30:01.096208482 (518h1m0.967291527s) [upload=454470]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:30:01.096215836 (518h1m0.967298881s) [upload=454470]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:30:01.346408880 (518h1m1.217491925s) [seg_pos={0 0}, upload=454470]: Begin upload segment... Dec 27 05:30:01.346455208 (518h1m1.217538253s) [seg_pos={0 0}, upload=454470]: Done begin upload segment. Dec 27 05:30:01.346485606 (518h1m1.217568651s) [noise=true , address=145.79.227.38:4979 , piece_id=MNHCR6FSXYZDPYQGJ4WFWQCWPT7KVXJ2YAYFEU74LPSSPNIJPNPA, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.346522737 (518h1m1.217605782s) [noise=true , address=77.90.31.13:4629 , piece_id=66IKZOIC7VLBVFHXYIO5MBLUQTH7KR45Y53KS4XPVUU5RZQA3MVQ, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.347486802 (518h1m1.218569837s) [noise=true , address=77.90.31.11:4554 , piece_id=SHGQTDSCVYRMOSFQ2JYID6OWPWBGZZFMEIAN6RTGZMUTXCD3AM5A, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.347852719 (518h1m1.218935764s) [noise=true , address=45.91.116.11:4117 , piece_id=UKDMFENDSRDA66VIEINHZ2JZWBH4LTGOBEZHIL5MSHESVFCQILQQ, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.348187606 (518h1m1.219270652s) [noise=true , address=145.79.227.12:4765 , piece_id=3SRRIXJCLF4F4RFH24U2SZRQJC3SLEI5GG6KWKUSAAZYSKKQZUBQ, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.348512766 (518h1m1.219595811s) [noise=true , address=145.79.227.31:4798 , piece_id=KDRCZFPGWSWOIPYINLFH65H66377GAEAT6AZSHMMQ2DKADLGFQRA, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.348948485 (518h1m1.22003152s) [noise=true , address=145.79.227.49:4338 , piece_id=HRWTDWWDUC3OIZDVZ2BTCLAKLYRTXHBN5PNBXC4UV6LFIR7LUTQA, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.349159247 (518h1m1.220242292s) [noise=true , address=145.79.227.6:4259 , piece_id=QUZMCZKJ4MRAWAPZNXNIUIOINY5FAKCS4AJ774UV2AOL4SB4ZRCA, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.349480269 (518h1m1.220563354s) [noise=true , address=145.79.227.35:4482 , piece_id=5S7XONVQR3XYK77V2ZNZAQV6XT5X7BZKHPCA5VOIE4JUKR666GUQ, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.350011821 (518h1m1.221094866s) [noise=true , address=145.79.227.16:4912 , piece_id=VPKG5MRTTH73GTZ7KZN3MRGNB4LDE5SQP6Y37BSWNRDBEMIJBZLQ, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.350154783 (518h1m1.221237828s) [noise=true , address=145.79.227.28:4725 , piece_id=U3MQN4Z6CCDU4PRYFHXWKSZVWDDN5QQEAMMFT6KXN3XQ5EJBYFAA, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.350632703 (518h1m1.221715748s) [noise=true , address=145.79.227.9:4366 , piece_id=VUA3ZPNQSPMN3KJ62MH6MCBAB6WHZRBZEORW5R73JBOEWJYORZAQ, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.351096746 (518h1m1.222179791s) [noise=true , address=77.90.31.10:4518 , piece_id=OQJRZTXUWHAEIHG2RJIDXY46UTQ22NIFALGMYS2VBWNTKTPMMESA, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.351932588 (518h1m1.223015633s) [noise=true , address=145.79.227.47:4267 , piece_id=YITS5MVNX4FD45PK5AHUBLO2S3MZC2IGKBMQIKC34I3SL2HYWRKA, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.352490671 (518h1m1.223573716s) [noise=true , address=45.91.116.10:4082 , piece_id=4WGFMS2NKFJVN3MZRZSBGBLCMZLZCVW4HBHXHIIQELTMRKDRIKLQ, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:01.353072419 (518h1m1.224155464s) [noise=true , address=45.91.116.7:4010 , piece_id=TW23OVV5PEKUTGQJUXUMBVMMR36LMD3AIPJ5A5C5SCMTWBHP23YA, seg_pos={0 0}, upload=454470]: Uploading piece... Dec 27 05:30:03.022348599 (518h1m2.893431654s) [upload=454470]: Next returned nil segment Dec 27 05:30:03.022363837 (518h1m2.893446882s) [upload=454470]: Waiting for error group managing segments... Dec 27 05:30:03.469558869 (518h1m3.340641914s) [noise=true , address=145.79.227.9:4366 , piece_id=VUA3ZPNQSPMN3KJ62MH6MCBAB6WHZRBZEORW5R73JBOEWJYORZAQ, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.481290919 (518h1m3.352373974s) [noise=true , address=145.79.227.35:4482 , piece_id=5S7XONVQR3XYK77V2ZNZAQV6XT5X7BZKHPCA5VOIE4JUKR666GUQ, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.482487377 (518h1m3.353570422s) [noise=true , address=145.79.227.38:4979 , piece_id=MNHCR6FSXYZDPYQGJ4WFWQCWPT7KVXJ2YAYFEU74LPSSPNIJPNPA, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.490421064 (518h1m3.361504109s) [noise=true , address=145.79.227.6:4259 , piece_id=QUZMCZKJ4MRAWAPZNXNIUIOINY5FAKCS4AJ774UV2AOL4SB4ZRCA, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.492416974 (518h1m3.36350005s) [noise=true , address=145.79.227.31:4798 , piece_id=KDRCZFPGWSWOIPYINLFH65H66377GAEAT6AZSHMMQ2DKADLGFQRA, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.493111848 (518h1m3.364194893s) [noise=true , address=145.79.227.49:4338 , piece_id=HRWTDWWDUC3OIZDVZ2BTCLAKLYRTXHBN5PNBXC4UV6LFIR7LUTQA, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.509483723 (518h1m3.380566788s) [noise=true , address=145.79.227.12:4765 , piece_id=3SRRIXJCLF4F4RFH24U2SZRQJC3SLEI5GG6KWKUSAAZYSKKQZUBQ, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.516800175 (518h1m3.38788323s) [noise=true , address=145.79.227.16:4912 , piece_id=VPKG5MRTTH73GTZ7KZN3MRGNB4LDE5SQP6Y37BSWNRDBEMIJBZLQ, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.518153161 (518h1m3.389236206s) [noise=true , address=145.79.227.47:4267 , piece_id=YITS5MVNX4FD45PK5AHUBLO2S3MZC2IGKBMQIKC34I3SL2HYWRKA, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.542361138 (518h1m3.413444183s) [noise=true , address=145.79.227.28:4725 , piece_id=U3MQN4Z6CCDU4PRYFHXWKSZVWDDN5QQEAMMFT6KXN3XQ5EJBYFAA, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.660256653 (518h1m3.531339708s) [noise=true , address=77.90.31.13:4629 , piece_id=66IKZOIC7VLBVFHXYIO5MBLUQTH7KR45Y53KS4XPVUU5RZQA3MVQ, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.754264020 (518h1m3.625347066s) [noise=true , address=45.91.116.10:4082 , piece_id=4WGFMS2NKFJVN3MZRZSBGBLCMZLZCVW4HBHXHIIQELTMRKDRIKLQ, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.780453229 (518h1m3.651536274s) [noise=true , address=77.90.31.10:4518 , piece_id=OQJRZTXUWHAEIHG2RJIDXY46UTQ22NIFALGMYS2VBWNTKTPMMESA, seg_pos={0 0}, upload=454470]: Done uploading piece. err: <nil> Dec 27 05:30:03.780490309 (518h1m3.651573355s) [seg_pos={0 0}, upload=454470]: Segment reached optimal threshold of 13 pieces. Dec 27 05:30:03.780852670 (518h1m3.651935715s) [noise=true , address=45.91.116.7:4010 , piece_id=TW23OVV5PEKUTGQJUXUMBVMMR36LMD3AIPJ5A5C5SCMTWBHP23YA, seg_pos={0 0}, upload=454470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 05:30:03.780863049 (518h1m3.651946105s) [noise=true , address=45.91.116.7:4010 , piece_id=TW23OVV5PEKUTGQJUXUMBVMMR36LMD3AIPJ5A5C5SCMTWBHP23YA, seg_pos={0 0}, upload=454470]: Long tail cancellation due to optimal threshold Dec 27 05:30:03.781063722 (518h1m3.652146797s) [noise=true , address=77.90.31.11:4554 , piece_id=SHGQTDSCVYRMOSFQ2JYID6OWPWBGZZFMEIAN6RTGZMUTXCD3AM5A, seg_pos={0 0}, upload=454470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 27 05:30:03.781087446 (518h1m3.652170491s) [noise=true , address=77.90.31.11:4554 , piece_id=SHGQTDSCVYRMOSFQ2JYID6OWPWBGZZFMEIAN6RTGZMUTXCD3AM5A, seg_pos={0 0}, upload=454470]: Long tail cancellation due to optimal threshold Dec 27 05:30:03.781175254 (518h1m3.652258439s) [noise=true , address=45.91.116.11:4117 , piece_id=UKDMFENDSRDA66VIEINHZ2JZWBH4LTGOBEZHIL5MSHESVFCQILQQ, seg_pos={0 0}, upload=454470]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:56154->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:56154->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56154->45.91.116.11:4117: use of closed network connection Dec 27 05:30:03.781205191 (518h1m3.652288236s) [noise=true , address=45.91.116.11:4117 , piece_id=UKDMFENDSRDA66VIEINHZ2JZWBH4LTGOBEZHIL5MSHESVFCQILQQ, seg_pos={0 0}, upload=454470]: Long tail cancellation due to optimal threshold Dec 27 05:30:03.781230248 (518h1m3.652313293s) [seg_pos={0 0}, upload=454470]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:30:03.781554927 (518h1m3.652637982s) [upload=454470]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:30:03.781589803 (518h1m3.652672869s) [upload=454470]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:30:04.084729256 (518h1m3.955812311s) [upload=454470]: Done uploading segments... Dec 27 05:30:04.084757731 (518h1m3.955840776s) [upload=454470]: Upload finished. err: <nil> Duration: 30.005704043s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.645820495s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.645842597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..................................X...........] duration: 50.392693187s trace: https://jaeger.dev.storj.io/trace/63f68f1606b74937 FAIL (2025-12-27 05:30:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA) context deadline exceeded; piecestore: (Node ID: 168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx, Piece ID: JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ) context deadline exceeded ======LOGS====== Dec 27 05:20:06.192886534 (517h51m6.063969569s) [noise=true , address=145.79.227.2:4136 , piece_id=TEYF4XZLEEWODAH76AW3UHKUPTB4ULIC4JQTTWX77TKTQVBH7Y5Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.193332814 (517h51m6.064415859s) [noise=true , address=193.149.164.143:4699 , piece_id=BKDTTBGO7KGZBNKEILXLO4YCXC3JJHGDFNNC4IT2SL4JQTZZ4EAA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.193645419 (517h51m6.064728465s) [noise=true , address=145.79.227.44:4225 , piece_id=LVV7XRDDMZ2OKNNDM6V46C2W3K2F5KYAQAC66BWYPZFEY42OPUHQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194011196 (517h51m6.065094242s) [noise=true , address=145.79.227.34:4938 , piece_id=BSL7K2IT3O7KF6B7HEQ3TKGYYC6K42RWYXVFH6BYXZOHT2TVCNQA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194331446 (517h51m6.065414481s) [noise=true , address=77.90.31.10:4536 , piece_id=LVBZWIMVH2OMOLHSEPXXSQCHNTYMRXYRN7KS43DKTMJ2OWITC45A, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.194790350 (517h51m6.065873395s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195112773 (517h51m6.066195818s) [noise=true , address=77.90.31.17:4804 , piece_id=JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195470736 (517h51m6.066553771s) [noise=true , address=145.79.227.2:4137 , piece_id=FZUUJKMD7F4SEZ6M6UIL3S7DGHKEJKWE45ZDVEFDWPE47S765FAQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.195772339 (517h51m6.066855384s) [noise=true , address=145.79.227.30:4766 , piece_id=HOC57KMPUPKIT35ILDWWNN5SHSBX2EKN5C6ZLHY2HWBKFZ2QXDBQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.196239610 (517h51m6.067322645s) [noise=true , address=77.90.31.19:4848 , piece_id=EZP26DQ7FLJYKTEGV5D74BJ7WKSB3HV2WUF3YCLQDUW6HQIFOCTA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.197528574 (517h51m6.068611609s) [noise=true , address=77.90.31.14:4673 , piece_id=PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.198047893 (517h51m6.069130928s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.198645551 (517h51m6.069728596s) [noise=true , address=145.79.227.15:4903 , piece_id=HITMQCCZLOT4JOZZURN25CFTNI5NWOPZY4P46MD6RIUEIZXMFUPA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199105326 (517h51m6.070188361s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199740606 (517h51m6.070823641s) [noise=true , address=145.79.227.10:4436 , piece_id=22RZSZO265VEXKQVTAUQYRT223VKQY7N4DNAP5NFAOK5DH44MQAQ, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:06.199909958 (517h51m6.070993003s) [noise=true , address=77.90.31.13:4647 , piece_id=GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA, seg_pos={0 0}, upload=454334]: Uploading piece... Dec 27 05:20:07.569793709 (517h51m7.440876764s) [upload=454334]: Next returned nil segment Dec 27 05:20:07.569833555 (517h51m7.44091662s) [upload=454334]: Waiting for error group managing segments... Dec 27 05:20:07.922714305 (517h51m7.79379735s) [noise=true , address=193.149.164.143:4699 , piece_id=BKDTTBGO7KGZBNKEILXLO4YCXC3JJHGDFNNC4IT2SL4JQTZZ4EAA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:07.943656403 (517h51m7.814739448s) [noise=true , address=64.49.13.140:4507 , piece_id=RAIYSJOLWFJK76ZFXIKT47AUSHNNJPTU7ACE2VVPGPNWDOUJG4VQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.060758757 (517h51m7.931841802s) [noise=true , address=145.79.227.10:4436 , piece_id=22RZSZO265VEXKQVTAUQYRT223VKQY7N4DNAP5NFAOK5DH44MQAQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.064833787 (517h51m7.935916842s) [noise=true , address=145.79.227.15:4903 , piece_id=HITMQCCZLOT4JOZZURN25CFTNI5NWOPZY4P46MD6RIUEIZXMFUPA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.067874817 (517h51m7.938957852s) [noise=true , address=145.79.227.2:4137 , piece_id=FZUUJKMD7F4SEZ6M6UIL3S7DGHKEJKWE45ZDVEFDWPE47S765FAQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.101063785 (517h51m7.97214682s) [noise=true , address=145.79.227.2:4136 , piece_id=TEYF4XZLEEWODAH76AW3UHKUPTB4ULIC4JQTTWX77TKTQVBH7Y5Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.281421209 (517h51m8.152504254s) [noise=true , address=77.90.31.13:4647 , piece_id=GN7SL4APURX6QMM6CCBGECD24C36YEXMSNMEKF2QNPHDNYFQRZLA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.292502911 (517h51m8.163585956s) [noise=true , address=77.90.31.19:4848 , piece_id=EZP26DQ7FLJYKTEGV5D74BJ7WKSB3HV2WUF3YCLQDUW6HQIFOCTA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.347820305 (517h51m8.21890335s) [noise=true , address=77.90.31.14:4673 , piece_id=PWTGSL4U5AEJQG6DAY2Z434KXOGQT72UZDHLWBW7A4M6CSBODAVQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.369437187 (517h51m8.240520222s) [noise=true , address=77.90.31.10:4536 , piece_id=LVBZWIMVH2OMOLHSEPXXSQCHNTYMRXYRN7KS43DKTMJ2OWITC45A, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.623303356 (517h51m8.494386401s) [noise=true , address=77.90.31.17:4804 , piece_id=JXICW5YYPFZBMOX5ZJV373URMJYKX23RS2V72OD464UV3X2XERZQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:08.970073139 (517h51m8.841156184s) [noise=true , address=145.79.227.44:4225 , piece_id=LVV7XRDDMZ2OKNNDM6V46C2W3K2F5KYAQAC66BWYPZFEY42OPUHQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.004503902 (517h51m8.875586947s) [noise=true , address=145.79.227.30:4766 , piece_id=HOC57KMPUPKIT35ILDWWNN5SHSBX2EKN5C6ZLHY2HWBKFZ2QXDBQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.064493524 (517h51m8.935576569s) [noise=true , address=145.79.227.34:4938 , piece_id=BSL7K2IT3O7KF6B7HEQ3TKGYYC6K42RWYXVFH6BYXZOHT2TVCNQA, seg_pos={0 0}, upload=454334]: Done uploading piece. err: <nil> Dec 27 05:20:09.064523371 (517h51m8.935606416s) [seg_pos={0 0}, upload=454334]: Segment reached optimal threshold of 14 pieces. Dec 27 05:20:09.064788426 (517h51m8.935871471s) [noise=true , address=64.49.13.169:4707 , piece_id=AIIKCSLONZRP6G2VLKZVS5XXA33RZKZEEKU5JKK34OQE55YZARCQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36702->64.49.13.169:4707: use of closed network connection Dec 27 05:20:09.064801250 (517h51m8.935884295s) [noise=true , address=64.49.13.169:4707 , piece_id=AIIKCSLONZRP6G2VLKZVS5XXA33RZKZEEKU5JKK34OQE55YZARCQ, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.064881923 (517h51m8.935964958s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.064899587 (517h51m8.935982632s) [noise=true , address=77.90.31.12:4606 , piece_id=EH5S5XGHOZ6VLAUPEJ54BZS5REYQBWLQGOFEXLVVQBURR3GOLI7Q, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065087855 (517h51m8.9361709s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.065098886 (517h51m8.936181921s) [noise=true , address=145.79.227.40:4053 , piece_id=BGP4WAM7ALWKD2FW7AGYGO2CZRKG4ULIDER5PC5WXK4ABXRPKU4Q, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065174411 (517h51m8.936257476s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 27 05:20:09.065186092 (517h51m8.936269138s) [noise=true , address=77.90.31.21:4909 , piece_id=UONLFXQRY47XXMPTGJKVMNRLIECBHZNUPCAKY5X3HQY2SCRM2ZWQ, seg_pos={0 0}, upload=454334]: Long tail cancellation due to optimal threshold Dec 27 05:20:09.065214106 (517h51m8.936297141s) [seg_pos={0 0}, upload=454334]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 05:20:09.065313125 (517h51m8.93639617s) [upload=454334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:20:09.065321080 (517h51m8.936404125s) [upload=454334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:20:09.374729791 (517h51m9.245812846s) [upload=454334]: Done uploading segments... Dec 27 05:20:09.374755670 (517h51m9.245838706s) [upload=454334]: Upload finished. err: <nil> Duration: 50.392712224s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)