Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [............X.......X....................................XXX] duration: 5.001814442s trace: https://jaeger.dev.storj.io/trace/2a66989535931432 FAIL (2025-12-29 09:47:32 UTC) orbiter warning: context deadline exceeded Duration: 5.00183992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X.........] duration: 5.665285456s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:47:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 5.665312988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 38.631802341s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:47:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 38.631829573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X.......X....................................XX] duration: 5.000706152s trace: https://jaeger.dev.storj.io/trace/75c682540ad433f5 FAIL (2025-12-29 09:47:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000728675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X........] duration: 5.973149156s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:47:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 5.973175005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X.......] duration: 6.108058698s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:46:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 6.108085259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 32.234852866s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:46:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 32.234877644s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.............X.......X....................................X] duration: 5.001532937s trace: https://jaeger.dev.storj.io/trace/58b9d77b35d49237 FAIL (2025-12-29 09:45:57 UTC) orbiter warning: context deadline exceeded Duration: 5.001552063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X......] duration: 14.367737815s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:45:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 14.367765166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 15.344905537s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:45:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 15.34493395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X.....] duration: 5.965150366s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:45:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 5.96517891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.............X.......X....................................] duration: 5.001037672s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:45:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00106298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 38.380891309s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:45:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 38.380906137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X....] duration: 6.006912147s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:45:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 6.006963164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X...] duration: 5.750843057s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:44:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 5.750872965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 40.995484074s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:44:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 40.9955084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.............X.......X...................................] duration: 5.000570904s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:44:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00059495s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.............X.......X..................................] duration: 3.501009441s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:43:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501035491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X..] duration: 5.894744083s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:43:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 5.894759764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 31.701251768s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:43:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 31.70128492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X.] duration: 5.950704001s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:43:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 5.950719018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.X] duration: 30.002754325s trace: https://jaeger.dev.storj.io/trace/5b169343b52e8cbc FAIL (2025-12-29 09:43:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q) context deadline exceeded ======LOGS====== ec 29 09:42:51.174030922 (570h13m51.045113957s) [upload=499931]: Starting upload Dec 29 09:42:51.174100414 (570h13m51.045183459s) [upload=499931]: Done starting upload Dec 29 09:42:51.174179384 (570h13m51.04526243s) [upload=499931]: Uploading segments... Dec 29 09:42:51.174331995 (570h13m51.04541503s) [upload=499931]: Got next segment. Inline: false Dec 29 09:42:51.174370087 (570h13m51.045453122s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:42:51.174374165 (570h13m51.0454572s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:42:51.420155282 (570h13m51.291238327s) [seg_pos={0 0}, upload=499931]: Begin upload segment... Dec 29 09:42:51.420195799 (570h13m51.291278834s) [seg_pos={0 0}, upload=499931]: Done begin upload segment. Dec 29 09:42:51.420219324 (570h13m51.291302359s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.420331948 (570h13m51.291414983s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421137191 (570h13m51.292220227s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.421661710 (570h13m51.292744755s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422184496 (570h13m51.293267531s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422526186 (570h13m51.293609232s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422862347 (570h13m51.293945382s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.422912813 (570h13m51.293995858s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423245567 (570h13m51.294328602s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423390352 (570h13m51.294473387s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423567369 (570h13m51.294650414s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423883321 (570h13m51.294966356s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.423969925 (570h13m51.29505296s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424189354 (570h13m51.295272389s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424521185 (570h13m51.29560422s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:51.424831526 (570h13m51.295914561s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Uploading piece... Dec 29 09:42:53.120356014 (570h13m52.991439049s) [upload=499931]: Next returned nil segment Dec 29 09:42:53.120370002 (570h13m52.991453047s) [upload=499931]: Waiting for error group managing segments... Dec 29 09:42:53.398889517 (570h13m53.269972552s) [noise=true , address=145.79.227.17:4972 , piece_id=RNEOZPUXZLPJ33UE6UXRAJ2T7JV332XFCQ2VVXNH56L5REEJNYLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.408806471 (570h13m53.279889506s) [noise=true , address=145.79.227.46:4601 , piece_id=WVDDTN2FNEWK4A7BZ4BZUKXFN34PAIJQRMR2PT3XQHZDBSBUQN2A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414186084 (570h13m53.285269119s) [noise=true , address=145.79.227.7:4294 , piece_id=67I5IYKCKLJIWPGEQJSKA7RXYVFDJD63NLVZSZQSPELZMFNJSYPQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414277638 (570h13m53.285360683s) [noise=true , address=145.79.227.41:4087 , piece_id=QITFXUMSUX3SD6ABPULILXGS6YASSC5XGIHFJ5IOYUZ7E634IMWA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.414803279 (570h13m53.285886314s) [noise=true , address=145.79.227.21:4473 , piece_id=W7DNZKR3GMRQE4Z2OZESLFQIBPS5UWWVO4SBFL7BBSI4YEE7M2DQ, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.428753232 (570h13m53.299836267s) [noise=true , address=145.79.227.6:4257 , piece_id=NZOBYCXDHTSYI4SVZNT6632KI3Y2CTCVAGKN7GEWIIOGE3LHAKIA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.431482309 (570h13m53.302565354s) [noise=true , address=145.79.227.26:4655 , piece_id=TKLRMZQTKHEIDQXL3R3DFH75GRL7XHSZZH67NOU6C4JZMJ7CDI3Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.433012182 (570h13m53.304095217s) [noise=true , address=145.79.227.42:4122 , piece_id=WAJPK4EGTYN2EQSAFC7VYA5SJM2BJRTQOF53UCKEG5VQSTZ5QITA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.550348138 (570h13m53.421431173s) [noise=true , address=145.79.227.19:4077 , piece_id=NCOC6LZE7JPP6AUQANXSR23GFAA3KPBAEJ4QKEMKCOPVRXDOBHMA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.679105663 (570h13m53.550188698s) [noise=true , address=77.90.31.19:4841 , piece_id=TGEJGUVM2ARBRITX62CC7UHC4BGHTZPRLTB3NX6TLHUBQCKAC5JA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.708615594 (570h13m53.57969863s) [noise=true , address=45.91.116.9:4263 , piece_id=WBWLJEA53CK4MYIP2LEIM3WNUO2YYQCDBGRP7TUS2NC52RQHKZBA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.725538438 (570h13m53.596621473s) [noise=true , address=45.91.116.11:4119 , piece_id=KKZPBNI7GH4BQRLFU4BLRKRVP42WBDDIEY3DH3QXFUA5X6MXOS6A, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869873485 (570h13m53.74095653s) [noise=true , address=45.91.116.7:4009 , piece_id=XCT3OBCUDL3BSENCOHVK3YRMQ2FO7OCKHXL2ROBP2YYQM3SKX53Q, seg_pos={0 0}, upload=499931]: Done uploading piece. err: <nil> Dec 29 09:42:53.869897431 (570h13m53.740980476s) [seg_pos={0 0}, upload=499931]: Segment reached optimal threshold of 13 pieces. Dec 29 09:42:53.870143119 (570h13m53.741226174s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870151094 (570h13m53.741234149s) [noise=true , address=145.79.227.29:4943 , piece_id=Q2WZQMQPBWZQR3QRLPA7OKFF26DSS6KKA7YZZF3JAFA3I3UR2KLA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870287543 (570h13m53.741370578s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43806->45.91.116.13:4153: use of closed network connection Dec 29 09:42:53.870313333 (570h13m53.741396378s) [noise=true , address=45.91.116.13:4153 , piece_id=LWPYAHFMWHHMEYWEX2GVQPELOAAUTYNJMBHORCX5KUT4HZNA7MSA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870550605 (570h13m53.74163366s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 09:42:53.870560543 (570h13m53.741643589s) [noise=true , address=77.90.31.10:4518 , piece_id=EU33UKKIMCFTLULXBJMA2PX3Z26EESNHHG6GPO5MTXZAPJANR5YA, seg_pos={0 0}, upload=499931]: Long tail cancellation due to optimal threshold Dec 29 09:42:53.870584389 (570h13m53.741667424s) [seg_pos={0 0}, upload=499931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:42:53.870667838 (570h13m53.741750893s) [upload=499931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:42:53.870674450 (570h13m53.741757495s) [upload=499931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:42:54.158078053 (570h13m54.029161098s) [upload=499931]: Done uploading segments... Dec 29 09:42:54.158097159 (570h13m54.029180194s) [upload=499931]: Upload finished. err: <nil> Duration: 30.002774453s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X.............X.......X.................................] duration: 3.516273125s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:43:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516296017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 38.905038254s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:43:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 38.905066266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X....X.............X.......X................................] duration: 3.575834029s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:42:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.575861612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [XX....X.............X.......X...............................] duration: 3.531542247s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:42:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531563628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X.] duration: 5.646901594s trace: https://jaeger.dev.storj.io/trace/33e6f3b941e93db FAIL (2025-12-29 09:41:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: YRUSOCMNMWT3FUZGOZR4ZGED7L42IYFKQW2XVI4YF4VDVRAN57RA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: ESKVM2NZYFABK6NNNN4TFUXIJ7TMYX6VAMWFO27NFG2EM5Z4P5SQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IZ2MUSYJOBGWOIU6LJBSNDFTHA6O5N42CR5IWLDCHDJLZJMID5TQ) context deadline exceeded ======LOGS====== Dec 29 09:40:59.764561161 (570h11m59.635644196s) [upload=499904]: Starting upload Dec 29 09:40:59.764633178 (570h11m59.635716213s) [upload=499904]: Done starting upload Dec 29 09:40:59.764735182 (570h11m59.635818217s) [upload=499904]: Uploading segments... Dec 29 09:40:59.764854730 (570h11m59.635937765s) [upload=499904]: Got next segment. Inline: false Dec 29 09:40:59.764920394 (570h11m59.636003429s) [upload=499904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:40:59.764925946 (570h11m59.636008981s) [upload=499904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:41:00.006631190 (570h11m59.877714235s) [seg_pos={0 0}, upload=499904]: Begin upload segment... Dec 29 09:41:00.006689431 (570h11m59.877772476s) [seg_pos={0 0}, upload=499904]: Done begin upload segment. Dec 29 09:41:00.006730729 (570h11m59.877813765s) [noise=true , address=145.79.227.39:4014 , piece_id=FE7FFVND7SLMXNY2Q37WTNYCFADHS6KM3MANH45WCESFGOMRSUSQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.006769393 (570h11m59.877852428s) [noise=true , address=77.90.31.14:4661 , piece_id=V55RSXBP73PXBVDDRISFF3MS5GZ46NUDPDKBYC3PQXEVXK2GTKZA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.007258534 (570h11m59.878341569s) [noise=true , address=145.79.227.24:4582 , piece_id=RFI3GANLM3ASK5J22HUEXIU2Y57XZLRFPOBAKUPSFJ7RKUSFFYRA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.007562754 (570h11m59.878645799s) [noise=true , address=77.90.31.21:4880 , piece_id=OM3MGIU7T4JG2G37HZ7EJ6MPMIYG4OINKDFPUK6JA37QR6DJZ5GQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.007838940 (570h11m59.878921975s) [noise=true , address=45.91.116.4:4940 , piece_id=5WAKCRXQYO6RGRQ5JWKPFO3YQTXBWN4SGH3BQOCR4FML25XVPITQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.008149561 (570h11m59.879232596s) [noise=true , address=77.90.31.13:4629 , piece_id=YRUSOCMNMWT3FUZGOZR4ZGED7L42IYFKQW2XVI4YF4VDVRAN57RA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.008440215 (570h11m59.87952325s) [noise=true , address=145.79.227.14:4840 , piece_id=MRLAD76UJ3JCFXRK52WOOMZAKA2W75HSHQJ5CX5VCHEJ73QBHGFQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.008711571 (570h11m59.879794606s) [noise=true , address=145.79.227.42:4122 , piece_id=55SXAV3BAPCLEOJ4HK35I26HKSLTEMBXGNFD3KXPUXGSIG6GXSIA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.009023255 (570h11m59.88010629s) [noise=true , address=145.79.227.52:4446 , piece_id=2Y7RCZW7IDAB4LVIDBLTAFCHDZLTLOL4NDL46R452B4JBHL3Y2FQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.010721479 (570h11m59.881804514s) [noise=true , address=77.90.31.19:4841 , piece_id=BGEDCA6NHEL4TKQCDMHZKBOGGHBZ5CNAO7KUUEJ4WEHFMPDR3GPA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.014440881 (570h11m59.885523917s) [noise=true , address=145.79.227.3:4151 , piece_id=XN7TEVWKPTNZGAIIAOY67LYTAJYUOJ7B2O6ZBLYTMLOBL5E4FDQA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.017941417 (570h11m59.889024452s) [noise=true , address=145.79.227.25:4619 , piece_id=7BVPVA2I2ORW6TPMHXITX6FZCQWL54T7DCKNBGVXNR6REVXAVS7A, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.019477332 (570h11m59.890560367s) [noise=true , address=45.91.116.5:4297 , piece_id=ESKVM2NZYFABK6NNNN4TFUXIJ7TMYX6VAMWFO27NFG2EM5Z4P5SQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.021870408 (570h11m59.892953443s) [noise=true , address=145.79.227.29:4942 , piece_id=MOOAT5FA4YBDPSEY2GUXAUTZUEQF4QWNT23ZDXCSKMGSDZL4AUWA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.022863800 (570h11m59.893946835s) [noise=true , address=145.79.227.36:4517 , piece_id=E3Z6DVYZ6J7FUCHXAMTWKW6B7JZWE6NSG5DYGALEVTC6BLUKZY6Q, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.025438122 (570h11m59.896521167s) [noise=true , address=45.91.116.13:4153 , piece_id=IZ2MUSYJOBGWOIU6LJBSNDFTHA6O5N42CR5IWLDCHDJLZJMID5TQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:01.713858164 (570h12m1.584941209s) [upload=499904]: Next returned nil segment Dec 29 09:41:01.713874274 (570h12m1.584957309s) [upload=499904]: Waiting for error group managing segments... Dec 29 09:41:01.998899547 (570h12m1.869982582s) [noise=true , address=145.79.227.42:4122 , piece_id=55SXAV3BAPCLEOJ4HK35I26HKSLTEMBXGNFD3KXPUXGSIG6GXSIA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:01.999095631 (570h12m1.870178686s) [noise=true , address=145.79.227.24:4582 , piece_id=RFI3GANLM3ASK5J22HUEXIU2Y57XZLRFPOBAKUPSFJ7RKUSFFYRA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.010220245 (570h12m1.88130328s) [noise=true , address=145.79.227.25:4619 , piece_id=7BVPVA2I2ORW6TPMHXITX6FZCQWL54T7DCKNBGVXNR6REVXAVS7A, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.012532918 (570h12m1.883615963s) [noise=true , address=145.79.227.36:4517 , piece_id=E3Z6DVYZ6J7FUCHXAMTWKW6B7JZWE6NSG5DYGALEVTC6BLUKZY6Q, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.018173609 (570h12m1.889256654s) [noise=true , address=145.79.227.14:4840 , piece_id=MRLAD76UJ3JCFXRK52WOOMZAKA2W75HSHQJ5CX5VCHEJ73QBHGFQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.021237072 (570h12m1.892320117s) [noise=true , address=145.79.227.52:4446 , piece_id=2Y7RCZW7IDAB4LVIDBLTAFCHDZLTLOL4NDL46R452B4JBHL3Y2FQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.021493941 (570h12m1.892576986s) [noise=true , address=145.79.227.39:4014 , piece_id=FE7FFVND7SLMXNY2Q37WTNYCFADHS6KM3MANH45WCESFGOMRSUSQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.024384775 (570h12m1.895467821s) [noise=true , address=145.79.227.3:4151 , piece_id=XN7TEVWKPTNZGAIIAOY67LYTAJYUOJ7B2O6ZBLYTMLOBL5E4FDQA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.200383816 (570h12m2.071466861s) [noise=true , address=77.90.31.13:4629 , piece_id=YRUSOCMNMWT3FUZGOZR4ZGED7L42IYFKQW2XVI4YF4VDVRAN57RA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.241658630 (570h12m2.112741675s) [noise=true , address=45.91.116.5:4297 , piece_id=ESKVM2NZYFABK6NNNN4TFUXIJ7TMYX6VAMWFO27NFG2EM5Z4P5SQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.287591423 (570h12m2.158674468s) [noise=true , address=45.91.116.13:4153 , piece_id=IZ2MUSYJOBGWOIU6LJBSNDFTHA6O5N42CR5IWLDCHDJLZJMID5TQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.360041369 (570h12m2.231124414s) [noise=true , address=45.91.116.4:4940 , piece_id=5WAKCRXQYO6RGRQ5JWKPFO3YQTXBWN4SGH3BQOCR4FML25XVPITQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.535533594 (570h12m2.40661664s) [noise=true , address=77.90.31.19:4841 , piece_id=BGEDCA6NHEL4TKQCDMHZKBOGGHBZ5CNAO7KUUEJ4WEHFMPDR3GPA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.535559815 (570h12m2.40664286s) [seg_pos={0 0}, upload=499904]: Segment reached optimal threshold of 13 pieces. Dec 29 09:41:02.535803489 (570h12m2.406886534s) [noise=true , address=77.90.31.14:4661 , piece_id=V55RSXBP73PXBVDDRISFF3MS5GZ46NUDPDKBYC3PQXEVXK2GTKZA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:59254->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:59254->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59254->77.90.31.14:4661: use of closed network connection Dec 29 09:41:02.535813748 (570h12m2.406896793s) [noise=true , address=77.90.31.14:4661 , piece_id=V55RSXBP73PXBVDDRISFF3MS5GZ46NUDPDKBYC3PQXEVXK2GTKZA, seg_pos={0 0}, upload=499904]: Long tail cancellation due to optimal threshold Dec 29 09:41:02.535994873 (570h12m2.407077908s) [noise=true , address=145.79.227.29:4942 , piece_id=MOOAT5FA4YBDPSEY2GUXAUTZUEQF4QWNT23ZDXCSKMGSDZL4AUWA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 09:41:02.536003870 (570h12m2.407086915s) [noise=true , address=145.79.227.29:4942 , piece_id=MOOAT5FA4YBDPSEY2GUXAUTZUEQF4QWNT23ZDXCSKMGSDZL4AUWA, seg_pos={0 0}, upload=499904]: Long tail cancellation due to optimal threshold Dec 29 09:41:02.536018798 (570h12m2.407101843s) [noise=true , address=77.90.31.21:4880 , piece_id=OM3MGIU7T4JG2G37HZ7EJ6MPMIYG4OINKDFPUK6JA37QR6DJZ5GQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 09:41:02.536032464 (570h12m2.407115509s) [noise=true , address=77.90.31.21:4880 , piece_id=OM3MGIU7T4JG2G37HZ7EJ6MPMIYG4OINKDFPUK6JA37QR6DJZ5GQ, seg_pos={0 0}, upload=499904]: Long tail cancellation due to optimal threshold Dec 29 09:41:02.536056129 (570h12m2.407139164s) [seg_pos={0 0}, upload=499904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:41:02.536148035 (570h12m2.40723108s) [upload=499904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:41:02.536155759 (570h12m2.407238804s) [upload=499904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:41:02.828698747 (570h12m2.699781782s) [upload=499904]: Done uploading segments... Dec 29 09:41:02.828722703 (570h12m2.699805748s) [upload=499904]: Upload finished. err: <nil> Duration: 5.646925589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 15.366876994s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:41:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 15.366902532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.X] duration: 30.014275981s trace: https://jaeger.dev.storj.io/trace/33e6f3b941e93db FAIL (2025-12-29 09:41:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: YRUSOCMNMWT3FUZGOZR4ZGED7L42IYFKQW2XVI4YF4VDVRAN57RA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: ESKVM2NZYFABK6NNNN4TFUXIJ7TMYX6VAMWFO27NFG2EM5Z4P5SQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IZ2MUSYJOBGWOIU6LJBSNDFTHA6O5N42CR5IWLDCHDJLZJMID5TQ) context deadline exceeded ======LOGS====== Dec 29 09:40:59.764561161 (570h11m59.635644196s) [upload=499904]: Starting upload Dec 29 09:40:59.764633178 (570h11m59.635716213s) [upload=499904]: Done starting upload Dec 29 09:40:59.764735182 (570h11m59.635818217s) [upload=499904]: Uploading segments... Dec 29 09:40:59.764854730 (570h11m59.635937765s) [upload=499904]: Got next segment. Inline: false Dec 29 09:40:59.764920394 (570h11m59.636003429s) [upload=499904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:40:59.764925946 (570h11m59.636008981s) [upload=499904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:41:00.006631190 (570h11m59.877714235s) [seg_pos={0 0}, upload=499904]: Begin upload segment... Dec 29 09:41:00.006689431 (570h11m59.877772476s) [seg_pos={0 0}, upload=499904]: Done begin upload segment. Dec 29 09:41:00.006730729 (570h11m59.877813765s) [noise=true , address=145.79.227.39:4014 , piece_id=FE7FFVND7SLMXNY2Q37WTNYCFADHS6KM3MANH45WCESFGOMRSUSQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.006769393 (570h11m59.877852428s) [noise=true , address=77.90.31.14:4661 , piece_id=V55RSXBP73PXBVDDRISFF3MS5GZ46NUDPDKBYC3PQXEVXK2GTKZA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.007258534 (570h11m59.878341569s) [noise=true , address=145.79.227.24:4582 , piece_id=RFI3GANLM3ASK5J22HUEXIU2Y57XZLRFPOBAKUPSFJ7RKUSFFYRA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.007562754 (570h11m59.878645799s) [noise=true , address=77.90.31.21:4880 , piece_id=OM3MGIU7T4JG2G37HZ7EJ6MPMIYG4OINKDFPUK6JA37QR6DJZ5GQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.007838940 (570h11m59.878921975s) [noise=true , address=45.91.116.4:4940 , piece_id=5WAKCRXQYO6RGRQ5JWKPFO3YQTXBWN4SGH3BQOCR4FML25XVPITQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.008149561 (570h11m59.879232596s) [noise=true , address=77.90.31.13:4629 , piece_id=YRUSOCMNMWT3FUZGOZR4ZGED7L42IYFKQW2XVI4YF4VDVRAN57RA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.008440215 (570h11m59.87952325s) [noise=true , address=145.79.227.14:4840 , piece_id=MRLAD76UJ3JCFXRK52WOOMZAKA2W75HSHQJ5CX5VCHEJ73QBHGFQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.008711571 (570h11m59.879794606s) [noise=true , address=145.79.227.42:4122 , piece_id=55SXAV3BAPCLEOJ4HK35I26HKSLTEMBXGNFD3KXPUXGSIG6GXSIA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.009023255 (570h11m59.88010629s) [noise=true , address=145.79.227.52:4446 , piece_id=2Y7RCZW7IDAB4LVIDBLTAFCHDZLTLOL4NDL46R452B4JBHL3Y2FQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.010721479 (570h11m59.881804514s) [noise=true , address=77.90.31.19:4841 , piece_id=BGEDCA6NHEL4TKQCDMHZKBOGGHBZ5CNAO7KUUEJ4WEHFMPDR3GPA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.014440881 (570h11m59.885523917s) [noise=true , address=145.79.227.3:4151 , piece_id=XN7TEVWKPTNZGAIIAOY67LYTAJYUOJ7B2O6ZBLYTMLOBL5E4FDQA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.017941417 (570h11m59.889024452s) [noise=true , address=145.79.227.25:4619 , piece_id=7BVPVA2I2ORW6TPMHXITX6FZCQWL54T7DCKNBGVXNR6REVXAVS7A, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.019477332 (570h11m59.890560367s) [noise=true , address=45.91.116.5:4297 , piece_id=ESKVM2NZYFABK6NNNN4TFUXIJ7TMYX6VAMWFO27NFG2EM5Z4P5SQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.021870408 (570h11m59.892953443s) [noise=true , address=145.79.227.29:4942 , piece_id=MOOAT5FA4YBDPSEY2GUXAUTZUEQF4QWNT23ZDXCSKMGSDZL4AUWA, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.022863800 (570h11m59.893946835s) [noise=true , address=145.79.227.36:4517 , piece_id=E3Z6DVYZ6J7FUCHXAMTWKW6B7JZWE6NSG5DYGALEVTC6BLUKZY6Q, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:00.025438122 (570h11m59.896521167s) [noise=true , address=45.91.116.13:4153 , piece_id=IZ2MUSYJOBGWOIU6LJBSNDFTHA6O5N42CR5IWLDCHDJLZJMID5TQ, seg_pos={0 0}, upload=499904]: Uploading piece... Dec 29 09:41:01.713858164 (570h12m1.584941209s) [upload=499904]: Next returned nil segment Dec 29 09:41:01.713874274 (570h12m1.584957309s) [upload=499904]: Waiting for error group managing segments... Dec 29 09:41:01.998899547 (570h12m1.869982582s) [noise=true , address=145.79.227.42:4122 , piece_id=55SXAV3BAPCLEOJ4HK35I26HKSLTEMBXGNFD3KXPUXGSIG6GXSIA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:01.999095631 (570h12m1.870178686s) [noise=true , address=145.79.227.24:4582 , piece_id=RFI3GANLM3ASK5J22HUEXIU2Y57XZLRFPOBAKUPSFJ7RKUSFFYRA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.010220245 (570h12m1.88130328s) [noise=true , address=145.79.227.25:4619 , piece_id=7BVPVA2I2ORW6TPMHXITX6FZCQWL54T7DCKNBGVXNR6REVXAVS7A, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.012532918 (570h12m1.883615963s) [noise=true , address=145.79.227.36:4517 , piece_id=E3Z6DVYZ6J7FUCHXAMTWKW6B7JZWE6NSG5DYGALEVTC6BLUKZY6Q, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.018173609 (570h12m1.889256654s) [noise=true , address=145.79.227.14:4840 , piece_id=MRLAD76UJ3JCFXRK52WOOMZAKA2W75HSHQJ5CX5VCHEJ73QBHGFQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.021237072 (570h12m1.892320117s) [noise=true , address=145.79.227.52:4446 , piece_id=2Y7RCZW7IDAB4LVIDBLTAFCHDZLTLOL4NDL46R452B4JBHL3Y2FQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.021493941 (570h12m1.892576986s) [noise=true , address=145.79.227.39:4014 , piece_id=FE7FFVND7SLMXNY2Q37WTNYCFADHS6KM3MANH45WCESFGOMRSUSQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.024384775 (570h12m1.895467821s) [noise=true , address=145.79.227.3:4151 , piece_id=XN7TEVWKPTNZGAIIAOY67LYTAJYUOJ7B2O6ZBLYTMLOBL5E4FDQA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.200383816 (570h12m2.071466861s) [noise=true , address=77.90.31.13:4629 , piece_id=YRUSOCMNMWT3FUZGOZR4ZGED7L42IYFKQW2XVI4YF4VDVRAN57RA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.241658630 (570h12m2.112741675s) [noise=true , address=45.91.116.5:4297 , piece_id=ESKVM2NZYFABK6NNNN4TFUXIJ7TMYX6VAMWFO27NFG2EM5Z4P5SQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.287591423 (570h12m2.158674468s) [noise=true , address=45.91.116.13:4153 , piece_id=IZ2MUSYJOBGWOIU6LJBSNDFTHA6O5N42CR5IWLDCHDJLZJMID5TQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.360041369 (570h12m2.231124414s) [noise=true , address=45.91.116.4:4940 , piece_id=5WAKCRXQYO6RGRQ5JWKPFO3YQTXBWN4SGH3BQOCR4FML25XVPITQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.535533594 (570h12m2.40661664s) [noise=true , address=77.90.31.19:4841 , piece_id=BGEDCA6NHEL4TKQCDMHZKBOGGHBZ5CNAO7KUUEJ4WEHFMPDR3GPA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: <nil> Dec 29 09:41:02.535559815 (570h12m2.40664286s) [seg_pos={0 0}, upload=499904]: Segment reached optimal threshold of 13 pieces. Dec 29 09:41:02.535803489 (570h12m2.406886534s) [noise=true , address=77.90.31.14:4661 , piece_id=V55RSXBP73PXBVDDRISFF3MS5GZ46NUDPDKBYC3PQXEVXK2GTKZA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:59254->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:59254->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59254->77.90.31.14:4661: use of closed network connection Dec 29 09:41:02.535813748 (570h12m2.406896793s) [noise=true , address=77.90.31.14:4661 , piece_id=V55RSXBP73PXBVDDRISFF3MS5GZ46NUDPDKBYC3PQXEVXK2GTKZA, seg_pos={0 0}, upload=499904]: Long tail cancellation due to optimal threshold Dec 29 09:41:02.535994873 (570h12m2.407077908s) [noise=true , address=145.79.227.29:4942 , piece_id=MOOAT5FA4YBDPSEY2GUXAUTZUEQF4QWNT23ZDXCSKMGSDZL4AUWA, seg_pos={0 0}, upload=499904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 09:41:02.536003870 (570h12m2.407086915s) [noise=true , address=145.79.227.29:4942 , piece_id=MOOAT5FA4YBDPSEY2GUXAUTZUEQF4QWNT23ZDXCSKMGSDZL4AUWA, seg_pos={0 0}, upload=499904]: Long tail cancellation due to optimal threshold Dec 29 09:41:02.536018798 (570h12m2.407101843s) [noise=true , address=77.90.31.21:4880 , piece_id=OM3MGIU7T4JG2G37HZ7EJ6MPMIYG4OINKDFPUK6JA37QR6DJZ5GQ, seg_pos={0 0}, upload=499904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 29 09:41:02.536032464 (570h12m2.407115509s) [noise=true , address=77.90.31.21:4880 , piece_id=OM3MGIU7T4JG2G37HZ7EJ6MPMIYG4OINKDFPUK6JA37QR6DJZ5GQ, seg_pos={0 0}, upload=499904]: Long tail cancellation due to optimal threshold Dec 29 09:41:02.536056129 (570h12m2.407139164s) [seg_pos={0 0}, upload=499904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:41:02.536148035 (570h12m2.40723108s) [upload=499904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:41:02.536155759 (570h12m2.407238804s) [upload=499904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:41:02.828698747 (570h12m2.699781782s) [upload=499904]: Done uploading segments... Dec 29 09:41:02.828722703 (570h12m2.699805748s) [upload=499904]: Upload finished. err: <nil> Duration: 30.014309034s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.XX....X.............X.......X..............................] duration: 3.616422015s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:41:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.616451641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 42.690687552s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:40:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 42.69071718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X.] duration: 5.991384331s trace: https://jaeger.dev.storj.io/trace/5b7f32cc40ccfb03 FAIL (2025-12-29 09:40:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 09:39:49.763089962 (570h10m49.634173007s) [upload=499889]: Starting upload Dec 29 09:39:49.763162059 (570h10m49.634245094s) [upload=499889]: Done starting upload Dec 29 09:39:49.763308478 (570h10m49.634391523s) [upload=499889]: Uploading segments... Dec 29 09:39:49.763449045 (570h10m49.63453209s) [upload=499889]: Got next segment. Inline: false Dec 29 09:39:49.763489202 (570h10m49.634572247s) [upload=499889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:39:49.763494612 (570h10m49.634577647s) [upload=499889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:39:50.006840391 (570h10m49.877923436s) [seg_pos={0 0}, upload=499889]: Begin upload segment... Dec 29 09:39:50.006890385 (570h10m49.87797342s) [seg_pos={0 0}, upload=499889]: Done begin upload segment. Dec 29 09:39:50.006915624 (570h10m49.877998669s) [noise=true , address=77.90.31.19:4841 , piece_id=6JD65LWCN3QKIFVOZZGR4F4RRS5VH6NDKLH24B2NDTVYUBXBJATA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.008464563 (570h10m49.879547608s) [noise=true , address=145.79.227.5:4223 , piece_id=ZQBZSGEHD27HBCJ23IIZFV7PRKP4FUN5WISN6BXEEJZ6B6EDGUMA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.009223749 (570h10m49.880306784s) [noise=true , address=145.79.227.47:4266 , piece_id=IZZ7TKLV3WSLQNUGHLV3HNORLXZ4ZU47ZOTEXPXF5TGU3DWIRMVA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.009593944 (570h10m49.880676989s) [noise=true , address=77.90.31.16:4735 , piece_id=JR6RXOFME6QNGCS6YJAWLO2CQDXPNW3BANMBZA2O3NDRWX724IZA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.011000692 (570h10m49.882083737s) [noise=true , address=45.91.116.12:4226 , piece_id=X3IGVUZA6ZZ6HGBM65A364MLMMTSH6LHEH5JCLKHRXUJSLSNY5TQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.011089752 (570h10m49.882172797s) [noise=true , address=145.79.227.37:4566 , piece_id=WZECZCZIKF65IJEBQZCDDXXVQ3ID4YIC2QT3Z47A5DEY6PKVRKHQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.012304384 (570h10m49.883387429s) [noise=true , address=145.79.227.38:4979 , piece_id=TKRUOXADSAJG2W65SF5A2SFY27OQHP3NTJBJ5ZSU7O3D5SOSZTYQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.012557026 (570h10m49.883640061s) [noise=true , address=45.91.116.8:4046 , piece_id=TJWD2OZRZK5TQYKARQPVH53MAU2YT3N77IVO6Z2DGHCSRRGUHMOA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.013485193 (570h10m49.884568228s) [noise=true , address=45.91.116.6:4974 , piece_id=4LKFAVMDY245UXDHROKJCIW5B543ERRW7PXSS7QWCPX4RBIRCKUA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.013934598 (570h10m49.885017634s) [noise=true , address=77.90.31.11:4553 , piece_id=IVSSEIQY4HLUR6X5PVQOEYNVJRP3IGOSEEJTCLG5L2UWFHAPXPHQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.014800798 (570h10m49.885883833s) [noise=true , address=145.79.227.2:4115 , piece_id=ZOURLMG7W4ST7VJ3NYNKDSA753W3HLVOQAMLBJ2HGJ2I4P3MUYPA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.015322572 (570h10m49.886405607s) [noise=true , address=45.91.116.11:4119 , piece_id=EPKDOIACMYD43K5CHQS4EXBCPJT5OBXRTCJGBQHU4TWYXR4XAQBQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.016078121 (570h10m49.887161156s) [noise=true , address=145.79.227.26:4654 , piece_id=JFZYGKPFMUHBAS7ERJAWD5YEDNHHCC42UERGF5WYQB6WML5ZDLLA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.018060534 (570h10m49.889143569s) [noise=true , address=145.79.227.41:4087 , piece_id=5YNBRO4MR7JOJTNLU57QW35NF7GFL4PIEBB2FZ6T4IIOYDVHGMBA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.021556612 (570h10m49.892639657s) [noise=true , address=45.91.116.13:4153 , piece_id=HLTJVDJJAC6J5KLEEBEM42BQTC2F2DUN3KD3BSOTFGGVCQFUA5LQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.024490338 (570h10m49.895573373s) [noise=true , address=145.79.227.35:4482 , piece_id=PJHPKQ2BJ3JJUCKBVWK7ONFTT5OFRQOIFJUMLK6X4GMSNF2IZW5Q, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:51.739256917 (570h10m51.610339952s) [upload=499889]: Next returned nil segment Dec 29 09:39:51.739269742 (570h10m51.610352777s) [upload=499889]: Waiting for error group managing segments... Dec 29 09:39:52.011784644 (570h10m51.882867679s) [noise=true , address=145.79.227.26:4654 , piece_id=JFZYGKPFMUHBAS7ERJAWD5YEDNHHCC42UERGF5WYQB6WML5ZDLLA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.013053169 (570h10m51.884136204s) [noise=true , address=145.79.227.41:4087 , piece_id=5YNBRO4MR7JOJTNLU57QW35NF7GFL4PIEBB2FZ6T4IIOYDVHGMBA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.016570246 (570h10m51.887653281s) [noise=true , address=145.79.227.38:4979 , piece_id=TKRUOXADSAJG2W65SF5A2SFY27OQHP3NTJBJ5ZSU7O3D5SOSZTYQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.017573447 (570h10m51.888656482s) [noise=true , address=145.79.227.2:4115 , piece_id=ZOURLMG7W4ST7VJ3NYNKDSA753W3HLVOQAMLBJ2HGJ2I4P3MUYPA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.023207976 (570h10m51.894291021s) [noise=true , address=145.79.227.37:4566 , piece_id=WZECZCZIKF65IJEBQZCDDXXVQ3ID4YIC2QT3Z47A5DEY6PKVRKHQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.036480479 (570h10m51.907563514s) [noise=true , address=145.79.227.47:4266 , piece_id=IZZ7TKLV3WSLQNUGHLV3HNORLXZ4ZU47ZOTEXPXF5TGU3DWIRMVA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.047046307 (570h10m51.918129343s) [noise=true , address=145.79.227.5:4223 , piece_id=ZQBZSGEHD27HBCJ23IIZFV7PRKP4FUN5WISN6BXEEJZ6B6EDGUMA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.292191623 (570h10m52.163274678s) [noise=true , address=45.91.116.11:4119 , piece_id=EPKDOIACMYD43K5CHQS4EXBCPJT5OBXRTCJGBQHU4TWYXR4XAQBQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.317056300 (570h10m52.188139345s) [noise=true , address=45.91.116.13:4153 , piece_id=HLTJVDJJAC6J5KLEEBEM42BQTC2F2DUN3KD3BSOTFGGVCQFUA5LQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.331636703 (570h10m52.202719738s) [noise=true , address=45.91.116.8:4046 , piece_id=TJWD2OZRZK5TQYKARQPVH53MAU2YT3N77IVO6Z2DGHCSRRGUHMOA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.336184833 (570h10m52.207267878s) [noise=true , address=45.91.116.6:4974 , piece_id=4LKFAVMDY245UXDHROKJCIW5B543ERRW7PXSS7QWCPX4RBIRCKUA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.376533754 (570h10m52.247616799s) [noise=true , address=45.91.116.12:4226 , piece_id=X3IGVUZA6ZZ6HGBM65A364MLMMTSH6LHEH5JCLKHRXUJSLSNY5TQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.544384318 (570h10m52.415467363s) [noise=true , address=77.90.31.16:4735 , piece_id=JR6RXOFME6QNGCS6YJAWLO2CQDXPNW3BANMBZA2O3NDRWX724IZA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.544408424 (570h10m52.415491479s) [seg_pos={0 0}, upload=499889]: Segment reached optimal threshold of 13 pieces. Dec 29 09:39:52.544712964 (570h10m52.415796009s) [noise=true , address=145.79.227.35:4482 , piece_id=PJHPKQ2BJ3JJUCKBVWK7ONFTT5OFRQOIFJUMLK6X4GMSNF2IZW5Q, seg_pos={0 0}, upload=499889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 09:39:52.544725197 (570h10m52.415808232s) [noise=true , address=145.79.227.35:4482 , piece_id=PJHPKQ2BJ3JJUCKBVWK7ONFTT5OFRQOIFJUMLK6X4GMSNF2IZW5Q, seg_pos={0 0}, upload=499889]: Long tail cancellation due to optimal threshold Dec 29 09:39:52.544789199 (570h10m52.415872274s) [noise=true , address=77.90.31.11:4553 , piece_id=IVSSEIQY4HLUR6X5PVQOEYNVJRP3IGOSEEJTCLG5L2UWFHAPXPHQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 09:39:52.544809128 (570h10m52.415892183s) [noise=true , address=77.90.31.11:4553 , piece_id=IVSSEIQY4HLUR6X5PVQOEYNVJRP3IGOSEEJTCLG5L2UWFHAPXPHQ, seg_pos={0 0}, upload=499889]: Long tail cancellation due to optimal threshold Dec 29 09:39:52.545069212 (570h10m52.416152258s) [noise=true , address=77.90.31.19:4841 , piece_id=6JD65LWCN3QKIFVOZZGR4F4RRS5VH6NDKLH24B2NDTVYUBXBJATA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:57520->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:57520->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57520->77.90.31.19:4841: use of closed network connection Dec 29 09:39:52.545090774 (570h10m52.416173839s) [noise=true , address=77.90.31.19:4841 , piece_id=6JD65LWCN3QKIFVOZZGR4F4RRS5VH6NDKLH24B2NDTVYUBXBJATA, seg_pos={0 0}, upload=499889]: Long tail cancellation due to optimal threshold Dec 29 09:39:52.545164354 (570h10m52.416247399s) [seg_pos={0 0}, upload=499889]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:39:52.545258292 (570h10m52.416341327s) [upload=499889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:39:52.545266148 (570h10m52.416349183s) [upload=499889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:39:52.845131678 (570h10m52.716214723s) [upload=499889]: Done uploading segments... Dec 29 09:39:52.845192394 (570h10m52.716275429s) [upload=499889]: Upload finished. err: <nil> Duration: 5.99139944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.X] duration: 30.003961559s trace: https://jaeger.dev.storj.io/trace/5b7f32cc40ccfb03 FAIL (2025-12-29 09:40:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 09:39:49.763089962 (570h10m49.634173007s) [upload=499889]: Starting upload Dec 29 09:39:49.763162059 (570h10m49.634245094s) [upload=499889]: Done starting upload Dec 29 09:39:49.763308478 (570h10m49.634391523s) [upload=499889]: Uploading segments... Dec 29 09:39:49.763449045 (570h10m49.63453209s) [upload=499889]: Got next segment. Inline: false Dec 29 09:39:49.763489202 (570h10m49.634572247s) [upload=499889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:39:49.763494612 (570h10m49.634577647s) [upload=499889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:39:50.006840391 (570h10m49.877923436s) [seg_pos={0 0}, upload=499889]: Begin upload segment... Dec 29 09:39:50.006890385 (570h10m49.87797342s) [seg_pos={0 0}, upload=499889]: Done begin upload segment. Dec 29 09:39:50.006915624 (570h10m49.877998669s) [noise=true , address=77.90.31.19:4841 , piece_id=6JD65LWCN3QKIFVOZZGR4F4RRS5VH6NDKLH24B2NDTVYUBXBJATA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.008464563 (570h10m49.879547608s) [noise=true , address=145.79.227.5:4223 , piece_id=ZQBZSGEHD27HBCJ23IIZFV7PRKP4FUN5WISN6BXEEJZ6B6EDGUMA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.009223749 (570h10m49.880306784s) [noise=true , address=145.79.227.47:4266 , piece_id=IZZ7TKLV3WSLQNUGHLV3HNORLXZ4ZU47ZOTEXPXF5TGU3DWIRMVA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.009593944 (570h10m49.880676989s) [noise=true , address=77.90.31.16:4735 , piece_id=JR6RXOFME6QNGCS6YJAWLO2CQDXPNW3BANMBZA2O3NDRWX724IZA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.011000692 (570h10m49.882083737s) [noise=true , address=45.91.116.12:4226 , piece_id=X3IGVUZA6ZZ6HGBM65A364MLMMTSH6LHEH5JCLKHRXUJSLSNY5TQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.011089752 (570h10m49.882172797s) [noise=true , address=145.79.227.37:4566 , piece_id=WZECZCZIKF65IJEBQZCDDXXVQ3ID4YIC2QT3Z47A5DEY6PKVRKHQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.012304384 (570h10m49.883387429s) [noise=true , address=145.79.227.38:4979 , piece_id=TKRUOXADSAJG2W65SF5A2SFY27OQHP3NTJBJ5ZSU7O3D5SOSZTYQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.012557026 (570h10m49.883640061s) [noise=true , address=45.91.116.8:4046 , piece_id=TJWD2OZRZK5TQYKARQPVH53MAU2YT3N77IVO6Z2DGHCSRRGUHMOA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.013485193 (570h10m49.884568228s) [noise=true , address=45.91.116.6:4974 , piece_id=4LKFAVMDY245UXDHROKJCIW5B543ERRW7PXSS7QWCPX4RBIRCKUA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.013934598 (570h10m49.885017634s) [noise=true , address=77.90.31.11:4553 , piece_id=IVSSEIQY4HLUR6X5PVQOEYNVJRP3IGOSEEJTCLG5L2UWFHAPXPHQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.014800798 (570h10m49.885883833s) [noise=true , address=145.79.227.2:4115 , piece_id=ZOURLMG7W4ST7VJ3NYNKDSA753W3HLVOQAMLBJ2HGJ2I4P3MUYPA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.015322572 (570h10m49.886405607s) [noise=true , address=45.91.116.11:4119 , piece_id=EPKDOIACMYD43K5CHQS4EXBCPJT5OBXRTCJGBQHU4TWYXR4XAQBQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.016078121 (570h10m49.887161156s) [noise=true , address=145.79.227.26:4654 , piece_id=JFZYGKPFMUHBAS7ERJAWD5YEDNHHCC42UERGF5WYQB6WML5ZDLLA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.018060534 (570h10m49.889143569s) [noise=true , address=145.79.227.41:4087 , piece_id=5YNBRO4MR7JOJTNLU57QW35NF7GFL4PIEBB2FZ6T4IIOYDVHGMBA, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.021556612 (570h10m49.892639657s) [noise=true , address=45.91.116.13:4153 , piece_id=HLTJVDJJAC6J5KLEEBEM42BQTC2F2DUN3KD3BSOTFGGVCQFUA5LQ, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:50.024490338 (570h10m49.895573373s) [noise=true , address=145.79.227.35:4482 , piece_id=PJHPKQ2BJ3JJUCKBVWK7ONFTT5OFRQOIFJUMLK6X4GMSNF2IZW5Q, seg_pos={0 0}, upload=499889]: Uploading piece... Dec 29 09:39:51.739256917 (570h10m51.610339952s) [upload=499889]: Next returned nil segment Dec 29 09:39:51.739269742 (570h10m51.610352777s) [upload=499889]: Waiting for error group managing segments... Dec 29 09:39:52.011784644 (570h10m51.882867679s) [noise=true , address=145.79.227.26:4654 , piece_id=JFZYGKPFMUHBAS7ERJAWD5YEDNHHCC42UERGF5WYQB6WML5ZDLLA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.013053169 (570h10m51.884136204s) [noise=true , address=145.79.227.41:4087 , piece_id=5YNBRO4MR7JOJTNLU57QW35NF7GFL4PIEBB2FZ6T4IIOYDVHGMBA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.016570246 (570h10m51.887653281s) [noise=true , address=145.79.227.38:4979 , piece_id=TKRUOXADSAJG2W65SF5A2SFY27OQHP3NTJBJ5ZSU7O3D5SOSZTYQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.017573447 (570h10m51.888656482s) [noise=true , address=145.79.227.2:4115 , piece_id=ZOURLMG7W4ST7VJ3NYNKDSA753W3HLVOQAMLBJ2HGJ2I4P3MUYPA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.023207976 (570h10m51.894291021s) [noise=true , address=145.79.227.37:4566 , piece_id=WZECZCZIKF65IJEBQZCDDXXVQ3ID4YIC2QT3Z47A5DEY6PKVRKHQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.036480479 (570h10m51.907563514s) [noise=true , address=145.79.227.47:4266 , piece_id=IZZ7TKLV3WSLQNUGHLV3HNORLXZ4ZU47ZOTEXPXF5TGU3DWIRMVA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.047046307 (570h10m51.918129343s) [noise=true , address=145.79.227.5:4223 , piece_id=ZQBZSGEHD27HBCJ23IIZFV7PRKP4FUN5WISN6BXEEJZ6B6EDGUMA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.292191623 (570h10m52.163274678s) [noise=true , address=45.91.116.11:4119 , piece_id=EPKDOIACMYD43K5CHQS4EXBCPJT5OBXRTCJGBQHU4TWYXR4XAQBQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.317056300 (570h10m52.188139345s) [noise=true , address=45.91.116.13:4153 , piece_id=HLTJVDJJAC6J5KLEEBEM42BQTC2F2DUN3KD3BSOTFGGVCQFUA5LQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.331636703 (570h10m52.202719738s) [noise=true , address=45.91.116.8:4046 , piece_id=TJWD2OZRZK5TQYKARQPVH53MAU2YT3N77IVO6Z2DGHCSRRGUHMOA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.336184833 (570h10m52.207267878s) [noise=true , address=45.91.116.6:4974 , piece_id=4LKFAVMDY245UXDHROKJCIW5B543ERRW7PXSS7QWCPX4RBIRCKUA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.376533754 (570h10m52.247616799s) [noise=true , address=45.91.116.12:4226 , piece_id=X3IGVUZA6ZZ6HGBM65A364MLMMTSH6LHEH5JCLKHRXUJSLSNY5TQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.544384318 (570h10m52.415467363s) [noise=true , address=77.90.31.16:4735 , piece_id=JR6RXOFME6QNGCS6YJAWLO2CQDXPNW3BANMBZA2O3NDRWX724IZA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: <nil> Dec 29 09:39:52.544408424 (570h10m52.415491479s) [seg_pos={0 0}, upload=499889]: Segment reached optimal threshold of 13 pieces. Dec 29 09:39:52.544712964 (570h10m52.415796009s) [noise=true , address=145.79.227.35:4482 , piece_id=PJHPKQ2BJ3JJUCKBVWK7ONFTT5OFRQOIFJUMLK6X4GMSNF2IZW5Q, seg_pos={0 0}, upload=499889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 09:39:52.544725197 (570h10m52.415808232s) [noise=true , address=145.79.227.35:4482 , piece_id=PJHPKQ2BJ3JJUCKBVWK7ONFTT5OFRQOIFJUMLK6X4GMSNF2IZW5Q, seg_pos={0 0}, upload=499889]: Long tail cancellation due to optimal threshold Dec 29 09:39:52.544789199 (570h10m52.415872274s) [noise=true , address=77.90.31.11:4553 , piece_id=IVSSEIQY4HLUR6X5PVQOEYNVJRP3IGOSEEJTCLG5L2UWFHAPXPHQ, seg_pos={0 0}, upload=499889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 09:39:52.544809128 (570h10m52.415892183s) [noise=true , address=77.90.31.11:4553 , piece_id=IVSSEIQY4HLUR6X5PVQOEYNVJRP3IGOSEEJTCLG5L2UWFHAPXPHQ, seg_pos={0 0}, upload=499889]: Long tail cancellation due to optimal threshold Dec 29 09:39:52.545069212 (570h10m52.416152258s) [noise=true , address=77.90.31.19:4841 , piece_id=6JD65LWCN3QKIFVOZZGR4F4RRS5VH6NDKLH24B2NDTVYUBXBJATA, seg_pos={0 0}, upload=499889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:57520->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:57520->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57520->77.90.31.19:4841: use of closed network connection Dec 29 09:39:52.545090774 (570h10m52.416173839s) [noise=true , address=77.90.31.19:4841 , piece_id=6JD65LWCN3QKIFVOZZGR4F4RRS5VH6NDKLH24B2NDTVYUBXBJATA, seg_pos={0 0}, upload=499889]: Long tail cancellation due to optimal threshold Dec 29 09:39:52.545164354 (570h10m52.416247399s) [seg_pos={0 0}, upload=499889]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:39:52.545258292 (570h10m52.416341327s) [upload=499889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:39:52.545266148 (570h10m52.416349183s) [upload=499889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:39:52.845131678 (570h10m52.716214723s) [upload=499889]: Done uploading segments... Dec 29 09:39:52.845192394 (570h10m52.716275429s) [upload=499889]: Upload finished. err: <nil> Duration: 30.003985355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 33.830724901s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:40:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 33.830754047s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..XX....X.............X.......X.............................] duration: 3.676244767s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:40:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.6763406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X.] duration: 6.131485306s trace: https://jaeger.dev.storj.io/trace/4f20540d30fabade FAIL (2025-12-29 09:39:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: Q6WQJME5C4I5ECNTL7VE2VAHZNG3M4M23TB6FHN3WNMHPBCVKFDQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: PPNJ4HV5R6A5CYHCM273ZAU3SLW26VJ7PEJAG5UT7UD7WXPNUAXA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: ORUMEJ7Q3D56JHFSYTFXDJR4CXGTWYTXAU42OSR575CLMTKBRUQA) context deadline exceeded ======LOGS====== 38:40.811467659 (570h9m40.682550724s) [upload=499870]: Starting upload Dec 29 09:38:40.811558371 (570h9m40.682641426s) [upload=499870]: Done starting upload Dec 29 09:38:40.811622814 (570h9m40.682705869s) [upload=499870]: Uploading segments... Dec 29 09:38:40.811853844 (570h9m40.682936899s) [upload=499870]: Got next segment. Inline: false Dec 29 09:38:40.811900262 (570h9m40.682983307s) [upload=499870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:38:40.811906344 (570h9m40.682989399s) [upload=499870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:38:41.061273680 (570h9m40.932356725s) [seg_pos={0 0}, upload=499870]: Begin upload segment... Dec 29 09:38:41.061320218 (570h9m40.932403253s) [seg_pos={0 0}, upload=499870]: Done begin upload segment. Dec 29 09:38:41.061333934 (570h9m40.932416969s) [noise=true , address=45.91.116.6:4975 , piece_id=B2G6LVR2DKDMNXVBP26TANV2PAQUO6KCW27LYXSYQGGL225DH55Q, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.061352970 (570h9m40.932436006s) [noise=true , address=77.90.31.12:4590 , piece_id=IDXBW435CJQWQB62CVBSNJDGTI2FYS6FRZ6CZJ5HFDE3763MEH5Q, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.061848604 (570h9m40.932931669s) [noise=true , address=145.79.227.38:4977 , piece_id=PKOCGQ6QQ76PTRVEJTIMKVMEUPDPDQPYBMTDAQ2DJZZEBSTSRLJA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.062885499 (570h9m40.933968534s) [noise=true , address=145.79.227.20:4437 , piece_id=Z4P2JMHUMGQPR5NRNZXVZGFTCGPMYQJNVPBN5S5VGOKCIGS4QBRQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.063183646 (570h9m40.934266691s) [noise=true , address=145.79.227.7:4294 , piece_id=MOHQWLXRFIVSIKOC2CI2HK55ERSPWWYWGY6Y7EHUXZBAK6QXFFQQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.063451346 (570h9m40.934534381s) [noise=true , address=145.79.227.13:4803 , piece_id=CJ2NV5T7H52RVHHPLOBDNLPFFXT57WTHXW7QTV2PDOLCOLN2GLBA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.063711722 (570h9m40.934794757s) [noise=true , address=145.79.227.37:4567 , piece_id=A537MLA33FE2JA4OL4HS3G4HSJGEBXGF4QI33XF254J4LDBEPYHQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.063973470 (570h9m40.935056505s) [noise=true , address=145.79.227.5:4222 , piece_id=XVF6PZDWNXVF3WTWWL2GA5JALETX64SDDSRXAODZWWPZVOXRPUTA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.064899744 (570h9m40.935982799s) [noise=true , address=145.79.227.51:4409 , piece_id=Q6WQJME5C4I5ECNTL7VE2VAHZNG3M4M23TB6FHN3WNMHPBCVKFDQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.065113171 (570h9m40.936196216s) [noise=true , address=145.79.227.42:4122 , piece_id=D3MSBTHXUWO7J7WTRPC2E7HIMSU3RHK6BRVTKMEFYD6SROQ52OPA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.066197164 (570h9m40.9372802s) [noise=true , address=145.79.227.6:4259 , piece_id=GYZLVXY3U5T3GTAR3RUNHUXYJCYTZVP7NIVBZTR7SGYJ3BSF7FIQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.066409779 (570h9m40.937492814s) [noise=true , address=45.91.116.8:4046 , piece_id=ORUMEJ7Q3D56JHFSYTFXDJR4CXGTWYTXAU42OSR575CLMTKBRUQA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.067456703 (570h9m40.938539738s) [noise=true , address=77.90.31.10:4519 , piece_id=PIONVDRPE4UPQAW2PYP6S5V676BBG3AZV7PD6KKPD3NLDIX6HDPQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.067747626 (570h9m40.938830661s) [noise=true , address=77.90.31.14:4663 , piece_id=PPNJ4HV5R6A5CYHCM273ZAU3SLW26VJ7PEJAG5UT7UD7WXPNUAXA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.068564302 (570h9m40.939647337s) [noise=true , address=45.91.116.11:4117 , piece_id=5Z3E6KAMFF3XH2J26U2SDEYZOWDDOLJC6OFA7TLPWPBRPT7YPI4Q, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.071152971 (570h9m40.942236036s) [noise=true , address=45.91.116.4:4940 , piece_id=777WV4QBAS5I5LIMOGAMLAD5TB3J7SAOE7GUSE6JACSEKDXB3BIQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:42.759938970 (570h9m42.631022025s) [upload=499870]: Next returned nil segment Dec 29 09:38:42.759980249 (570h9m42.631063284s) [upload=499870]: Waiting for error group managing segments... Dec 29 09:38:43.044325096 (570h9m42.915408141s) [noise=true , address=145.79.227.7:4294 , piece_id=MOHQWLXRFIVSIKOC2CI2HK55ERSPWWYWGY6Y7EHUXZBAK6QXFFQQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.048959962 (570h9m42.920042997s) [noise=true , address=145.79.227.38:4977 , piece_id=PKOCGQ6QQ76PTRVEJTIMKVMEUPDPDQPYBMTDAQ2DJZZEBSTSRLJA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.051987276 (570h9m42.923070311s) [noise=true , address=145.79.227.13:4803 , piece_id=CJ2NV5T7H52RVHHPLOBDNLPFFXT57WTHXW7QTV2PDOLCOLN2GLBA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.055341613 (570h9m42.926424648s) [noise=true , address=145.79.227.20:4437 , piece_id=Z4P2JMHUMGQPR5NRNZXVZGFTCGPMYQJNVPBN5S5VGOKCIGS4QBRQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.064942786 (570h9m42.936025821s) [noise=true , address=145.79.227.6:4259 , piece_id=GYZLVXY3U5T3GTAR3RUNHUXYJCYTZVP7NIVBZTR7SGYJ3BSF7FIQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.065040783 (570h9m42.936123828s) [noise=true , address=145.79.227.51:4409 , piece_id=Q6WQJME5C4I5ECNTL7VE2VAHZNG3M4M23TB6FHN3WNMHPBCVKFDQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.074917920 (570h9m42.946000955s) [noise=true , address=145.79.227.5:4222 , piece_id=XVF6PZDWNXVF3WTWWL2GA5JALETX64SDDSRXAODZWWPZVOXRPUTA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.080647450 (570h9m42.951730495s) [noise=true , address=145.79.227.37:4567 , piece_id=A537MLA33FE2JA4OL4HS3G4HSJGEBXGF4QI33XF254J4LDBEPYHQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.085032139 (570h9m42.956115174s) [noise=true , address=145.79.227.42:4122 , piece_id=D3MSBTHXUWO7J7WTRPC2E7HIMSU3RHK6BRVTKMEFYD6SROQ52OPA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.261956211 (570h9m43.133039246s) [noise=true , address=77.90.31.10:4519 , piece_id=PIONVDRPE4UPQAW2PYP6S5V676BBG3AZV7PD6KKPD3NLDIX6HDPQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.321035857 (570h9m43.192118893s) [noise=true , address=45.91.116.4:4940 , piece_id=777WV4QBAS5I5LIMOGAMLAD5TB3J7SAOE7GUSE6JACSEKDXB3BIQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.363100696 (570h9m43.234183731s) [noise=true , address=77.90.31.14:4663 , piece_id=PPNJ4HV5R6A5CYHCM273ZAU3SLW26VJ7PEJAG5UT7UD7WXPNUAXA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.447186939 (570h9m43.318269974s) [noise=true , address=45.91.116.8:4046 , piece_id=ORUMEJ7Q3D56JHFSYTFXDJR4CXGTWYTXAU42OSR575CLMTKBRUQA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.447206587 (570h9m43.318289622s) [seg_pos={0 0}, upload=499870]: Segment reached optimal threshold of 13 pieces. Dec 29 09:38:43.447384145 (570h9m43.31846718s) [noise=true , address=45.91.116.11:4117 , piece_id=5Z3E6KAMFF3XH2J26U2SDEYZOWDDOLJC6OFA7TLPWPBRPT7YPI4Q, seg_pos={0 0}, upload=499870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:36140->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:36140->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36140->45.91.116.11:4117: use of closed network connection Dec 29 09:38:43.447389996 (570h9m43.318473031s) [noise=true , address=45.91.116.11:4117 , piece_id=5Z3E6KAMFF3XH2J26U2SDEYZOWDDOLJC6OFA7TLPWPBRPT7YPI4Q, seg_pos={0 0}, upload=499870]: Long tail cancellation due to optimal threshold Dec 29 09:38:43.447531906 (570h9m43.318614941s) [noise=true , address=77.90.31.12:4590 , piece_id=IDXBW435CJQWQB62CVBSNJDGTI2FYS6FRZ6CZJ5HFDE3763MEH5Q, seg_pos={0 0}, upload=499870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 09:38:43.447537136 (570h9m43.318620171s) [noise=true , address=77.90.31.12:4590 , piece_id=IDXBW435CJQWQB62CVBSNJDGTI2FYS6FRZ6CZJ5HFDE3763MEH5Q, seg_pos={0 0}, upload=499870]: Long tail cancellation due to optimal threshold Dec 29 09:38:43.447793805 (570h9m43.31887687s) [noise=true , address=45.91.116.6:4975 , piece_id=B2G6LVR2DKDMNXVBP26TANV2PAQUO6KCW27LYXSYQGGL225DH55Q, seg_pos={0 0}, upload=499870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:56590->45.91.116.6:4975: use of closed network connection; conn closed: op Dec 29 09:38:43.447819885 (570h9m43.31890293s) [noise=true , address=45.91.116.6:4975 , piece_id=B2G6LVR2DKDMNXVBP26TANV2PAQUO6KCW27LYXSYQGGL225DH55Q, seg_pos={0 0}, upload=499870]: Long tail cancellation due to optimal threshold Dec 29 09:38:43.447846355 (570h9m43.31892939s) [seg_pos={0 0}, upload=499870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:38:43.447948579 (570h9m43.319031614s) [upload=499870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:38:43.447954190 (570h9m43.319037225s) [upload=499870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:38:43.735037533 (570h9m43.606120568s) [upload=499870]: Done uploading segments... Dec 29 09:38:43.735058412 (570h9m43.606141447s) [upload=499870]: Upload finished. err: <nil> Duration: 6.131540332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X...X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...X] duration: 30.002727426s trace: https://jaeger.dev.storj.io/trace/4f20540d30fabade FAIL (2025-12-29 09:39:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: Q6WQJME5C4I5ECNTL7VE2VAHZNG3M4M23TB6FHN3WNMHPBCVKFDQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: PPNJ4HV5R6A5CYHCM273ZAU3SLW26VJ7PEJAG5UT7UD7WXPNUAXA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: ORUMEJ7Q3D56JHFSYTFXDJR4CXGTWYTXAU42OSR575CLMTKBRUQA) context deadline exceeded ======LOGS====== 38:40.811467659 (570h9m40.682550724s) [upload=499870]: Starting upload Dec 29 09:38:40.811558371 (570h9m40.682641426s) [upload=499870]: Done starting upload Dec 29 09:38:40.811622814 (570h9m40.682705869s) [upload=499870]: Uploading segments... Dec 29 09:38:40.811853844 (570h9m40.682936899s) [upload=499870]: Got next segment. Inline: false Dec 29 09:38:40.811900262 (570h9m40.682983307s) [upload=499870]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:38:40.811906344 (570h9m40.682989399s) [upload=499870]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:38:41.061273680 (570h9m40.932356725s) [seg_pos={0 0}, upload=499870]: Begin upload segment... Dec 29 09:38:41.061320218 (570h9m40.932403253s) [seg_pos={0 0}, upload=499870]: Done begin upload segment. Dec 29 09:38:41.061333934 (570h9m40.932416969s) [noise=true , address=45.91.116.6:4975 , piece_id=B2G6LVR2DKDMNXVBP26TANV2PAQUO6KCW27LYXSYQGGL225DH55Q, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.061352970 (570h9m40.932436006s) [noise=true , address=77.90.31.12:4590 , piece_id=IDXBW435CJQWQB62CVBSNJDGTI2FYS6FRZ6CZJ5HFDE3763MEH5Q, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.061848604 (570h9m40.932931669s) [noise=true , address=145.79.227.38:4977 , piece_id=PKOCGQ6QQ76PTRVEJTIMKVMEUPDPDQPYBMTDAQ2DJZZEBSTSRLJA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.062885499 (570h9m40.933968534s) [noise=true , address=145.79.227.20:4437 , piece_id=Z4P2JMHUMGQPR5NRNZXVZGFTCGPMYQJNVPBN5S5VGOKCIGS4QBRQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.063183646 (570h9m40.934266691s) [noise=true , address=145.79.227.7:4294 , piece_id=MOHQWLXRFIVSIKOC2CI2HK55ERSPWWYWGY6Y7EHUXZBAK6QXFFQQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.063451346 (570h9m40.934534381s) [noise=true , address=145.79.227.13:4803 , piece_id=CJ2NV5T7H52RVHHPLOBDNLPFFXT57WTHXW7QTV2PDOLCOLN2GLBA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.063711722 (570h9m40.934794757s) [noise=true , address=145.79.227.37:4567 , piece_id=A537MLA33FE2JA4OL4HS3G4HSJGEBXGF4QI33XF254J4LDBEPYHQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.063973470 (570h9m40.935056505s) [noise=true , address=145.79.227.5:4222 , piece_id=XVF6PZDWNXVF3WTWWL2GA5JALETX64SDDSRXAODZWWPZVOXRPUTA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.064899744 (570h9m40.935982799s) [noise=true , address=145.79.227.51:4409 , piece_id=Q6WQJME5C4I5ECNTL7VE2VAHZNG3M4M23TB6FHN3WNMHPBCVKFDQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.065113171 (570h9m40.936196216s) [noise=true , address=145.79.227.42:4122 , piece_id=D3MSBTHXUWO7J7WTRPC2E7HIMSU3RHK6BRVTKMEFYD6SROQ52OPA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.066197164 (570h9m40.9372802s) [noise=true , address=145.79.227.6:4259 , piece_id=GYZLVXY3U5T3GTAR3RUNHUXYJCYTZVP7NIVBZTR7SGYJ3BSF7FIQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.066409779 (570h9m40.937492814s) [noise=true , address=45.91.116.8:4046 , piece_id=ORUMEJ7Q3D56JHFSYTFXDJR4CXGTWYTXAU42OSR575CLMTKBRUQA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.067456703 (570h9m40.938539738s) [noise=true , address=77.90.31.10:4519 , piece_id=PIONVDRPE4UPQAW2PYP6S5V676BBG3AZV7PD6KKPD3NLDIX6HDPQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.067747626 (570h9m40.938830661s) [noise=true , address=77.90.31.14:4663 , piece_id=PPNJ4HV5R6A5CYHCM273ZAU3SLW26VJ7PEJAG5UT7UD7WXPNUAXA, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.068564302 (570h9m40.939647337s) [noise=true , address=45.91.116.11:4117 , piece_id=5Z3E6KAMFF3XH2J26U2SDEYZOWDDOLJC6OFA7TLPWPBRPT7YPI4Q, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:41.071152971 (570h9m40.942236036s) [noise=true , address=45.91.116.4:4940 , piece_id=777WV4QBAS5I5LIMOGAMLAD5TB3J7SAOE7GUSE6JACSEKDXB3BIQ, seg_pos={0 0}, upload=499870]: Uploading piece... Dec 29 09:38:42.759938970 (570h9m42.631022025s) [upload=499870]: Next returned nil segment Dec 29 09:38:42.759980249 (570h9m42.631063284s) [upload=499870]: Waiting for error group managing segments... Dec 29 09:38:43.044325096 (570h9m42.915408141s) [noise=true , address=145.79.227.7:4294 , piece_id=MOHQWLXRFIVSIKOC2CI2HK55ERSPWWYWGY6Y7EHUXZBAK6QXFFQQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.048959962 (570h9m42.920042997s) [noise=true , address=145.79.227.38:4977 , piece_id=PKOCGQ6QQ76PTRVEJTIMKVMEUPDPDQPYBMTDAQ2DJZZEBSTSRLJA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.051987276 (570h9m42.923070311s) [noise=true , address=145.79.227.13:4803 , piece_id=CJ2NV5T7H52RVHHPLOBDNLPFFXT57WTHXW7QTV2PDOLCOLN2GLBA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.055341613 (570h9m42.926424648s) [noise=true , address=145.79.227.20:4437 , piece_id=Z4P2JMHUMGQPR5NRNZXVZGFTCGPMYQJNVPBN5S5VGOKCIGS4QBRQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.064942786 (570h9m42.936025821s) [noise=true , address=145.79.227.6:4259 , piece_id=GYZLVXY3U5T3GTAR3RUNHUXYJCYTZVP7NIVBZTR7SGYJ3BSF7FIQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.065040783 (570h9m42.936123828s) [noise=true , address=145.79.227.51:4409 , piece_id=Q6WQJME5C4I5ECNTL7VE2VAHZNG3M4M23TB6FHN3WNMHPBCVKFDQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.074917920 (570h9m42.946000955s) [noise=true , address=145.79.227.5:4222 , piece_id=XVF6PZDWNXVF3WTWWL2GA5JALETX64SDDSRXAODZWWPZVOXRPUTA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.080647450 (570h9m42.951730495s) [noise=true , address=145.79.227.37:4567 , piece_id=A537MLA33FE2JA4OL4HS3G4HSJGEBXGF4QI33XF254J4LDBEPYHQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.085032139 (570h9m42.956115174s) [noise=true , address=145.79.227.42:4122 , piece_id=D3MSBTHXUWO7J7WTRPC2E7HIMSU3RHK6BRVTKMEFYD6SROQ52OPA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.261956211 (570h9m43.133039246s) [noise=true , address=77.90.31.10:4519 , piece_id=PIONVDRPE4UPQAW2PYP6S5V676BBG3AZV7PD6KKPD3NLDIX6HDPQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.321035857 (570h9m43.192118893s) [noise=true , address=45.91.116.4:4940 , piece_id=777WV4QBAS5I5LIMOGAMLAD5TB3J7SAOE7GUSE6JACSEKDXB3BIQ, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.363100696 (570h9m43.234183731s) [noise=true , address=77.90.31.14:4663 , piece_id=PPNJ4HV5R6A5CYHCM273ZAU3SLW26VJ7PEJAG5UT7UD7WXPNUAXA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.447186939 (570h9m43.318269974s) [noise=true , address=45.91.116.8:4046 , piece_id=ORUMEJ7Q3D56JHFSYTFXDJR4CXGTWYTXAU42OSR575CLMTKBRUQA, seg_pos={0 0}, upload=499870]: Done uploading piece. err: <nil> Dec 29 09:38:43.447206587 (570h9m43.318289622s) [seg_pos={0 0}, upload=499870]: Segment reached optimal threshold of 13 pieces. Dec 29 09:38:43.447384145 (570h9m43.31846718s) [noise=true , address=45.91.116.11:4117 , piece_id=5Z3E6KAMFF3XH2J26U2SDEYZOWDDOLJC6OFA7TLPWPBRPT7YPI4Q, seg_pos={0 0}, upload=499870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:36140->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:36140->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36140->45.91.116.11:4117: use of closed network connection Dec 29 09:38:43.447389996 (570h9m43.318473031s) [noise=true , address=45.91.116.11:4117 , piece_id=5Z3E6KAMFF3XH2J26U2SDEYZOWDDOLJC6OFA7TLPWPBRPT7YPI4Q, seg_pos={0 0}, upload=499870]: Long tail cancellation due to optimal threshold Dec 29 09:38:43.447531906 (570h9m43.318614941s) [noise=true , address=77.90.31.12:4590 , piece_id=IDXBW435CJQWQB62CVBSNJDGTI2FYS6FRZ6CZJ5HFDE3763MEH5Q, seg_pos={0 0}, upload=499870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 09:38:43.447537136 (570h9m43.318620171s) [noise=true , address=77.90.31.12:4590 , piece_id=IDXBW435CJQWQB62CVBSNJDGTI2FYS6FRZ6CZJ5HFDE3763MEH5Q, seg_pos={0 0}, upload=499870]: Long tail cancellation due to optimal threshold Dec 29 09:38:43.447793805 (570h9m43.31887687s) [noise=true , address=45.91.116.6:4975 , piece_id=B2G6LVR2DKDMNXVBP26TANV2PAQUO6KCW27LYXSYQGGL225DH55Q, seg_pos={0 0}, upload=499870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:56590->45.91.116.6:4975: use of closed network connection; conn closed: op Dec 29 09:38:43.447819885 (570h9m43.31890293s) [noise=true , address=45.91.116.6:4975 , piece_id=B2G6LVR2DKDMNXVBP26TANV2PAQUO6KCW27LYXSYQGGL225DH55Q, seg_pos={0 0}, upload=499870]: Long tail cancellation due to optimal threshold Dec 29 09:38:43.447846355 (570h9m43.31892939s) [seg_pos={0 0}, upload=499870]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:38:43.447948579 (570h9m43.319031614s) [upload=499870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:38:43.447954190 (570h9m43.319037225s) [upload=499870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:38:43.735037533 (570h9m43.606120568s) [upload=499870]: Done uploading segments... Dec 29 09:38:43.735058412 (570h9m43.606141447s) [upload=499870]: Upload finished. err: <nil> Duration: 30.002748435s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...XX....X.............X.......X............................] duration: 3.592120101s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:39:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592143745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 34.775705933s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:38:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 34.775735699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X...X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X...] duration: 27.505707841s trace: https://jaeger.dev.storj.io/trace/5a2746f539f4d260 FAIL (2025-12-29 09:38:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ) context deadline exceeded ======LOGS====== Dec 29 09:36:01.579542678 (570h7m1.450625713s) [upload=499830]: Starting upload Dec 29 09:36:01.579633091 (570h7m1.450716136s) [upload=499830]: Done starting upload Dec 29 09:36:01.579811300 (570h7m1.450894345s) [upload=499830]: Uploading segments... Dec 29 09:36:01.579940636 (570h7m1.451023671s) [upload=499830]: Got next segment. Inline: false Dec 29 09:36:01.579990982 (570h7m1.451074047s) [upload=499830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:36:01.579998797 (570h7m1.451081852s) [upload=499830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:36:01.824871323 (570h7m1.695954388s) [seg_pos={0 0}, upload=499830]: Begin upload segment... Dec 29 09:36:01.824922631 (570h7m1.696005676s) [seg_pos={0 0}, upload=499830]: Done begin upload segment. Dec 29 09:36:01.824956285 (570h7m1.69603933s) [noise=true , address=77.90.31.17:4773 , piece_id=SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.825004567 (570h7m1.696087612s) [noise=true , address=145.79.227.42:4122 , piece_id=33VV7GFCQ4ZYLCJYQ6L3F6YPAZHMNUFKT5YLAGHIOBGZA2IOTYRQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.825568470 (570h7m1.696651506s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826057942 (570h7m1.697140998s) [noise=true , address=45.91.116.12:4227 , piece_id=2GLFHD4Z7GICBOS32P2BBVKFF5Y25XHVCV6BIIRCQDBINHYVGETA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826857184 (570h7m1.697940229s) [noise=true , address=145.79.227.43:4157 , piece_id=ED2JLQRCSTPZ36P2NHVCSM2GMYUHFD6BGSVAFJFEZYXSU5LMKIIA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826891199 (570h7m1.697974244s) [noise=true , address=45.91.116.14:4192 , piece_id=ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827342569 (570h7m1.698425614s) [noise=true , address=145.79.227.2:4113 , piece_id=7CGKHRRSMAIHUFTJ6TIDXQJ4SSQD6GX4AXWD2N6CVNH6DR5YXD3A, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827586253 (570h7m1.698669298s) [noise=true , address=145.79.227.37:4566 , piece_id=4TQ6OYIOF4OJ7V5NBBCVNGHNZDVKM2GVKM2P3V54GOJN4EZ54MYA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827944686 (570h7m1.699027721s) [noise=true , address=145.79.227.23:4547 , piece_id=VCQWI5U3AEF25V7T7TW643M3DIE6QWTAHBSFMKX6F7UB3TVQZIZQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.828307577 (570h7m1.699390632s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.828592890 (570h7m1.699675935s) [noise=true , address=145.79.227.26:4654 , piece_id=ESWH6NQWSGOZ7T42AOWMYZR2GMREMCBOODGOZ6JCBQ6AKZIZQMVQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.829222158 (570h7m1.70033492s) [noise=true , address=145.79.227.14:4838 , piece_id=QBDBP2XZ5PM5QGHOQEHBRAVDMO7XSPSPMM5J2CS2DEY33BME3D5A, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.829961175 (570h7m1.70104422s) [noise=true , address=145.79.227.25:4618 , piece_id=UYCIBF6LE4YCQVVDPR3F7CTX5AKH57RI2EHLY3MNJMJGZ5SGJ4BQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.831771393 (570h7m1.702854438s) [noise=true , address=145.79.227.39:4014 , piece_id=2T2CTDVIALPHXUQUB243BHOSJNQ6DDPNF3HD7UEGRFHAB4PSHMTA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.833759138 (570h7m1.704842173s) [noise=true , address=145.79.227.35:4482 , piece_id=VI4TIIPY6L7BCERHNS4PZVQSFJAGNAUPNQVCY4P767SCBPLYMTYQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.834168076 (570h7m1.705251111s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:03.535615254 (570h7m3.406698309s) [upload=499830]: Next returned nil segment Dec 29 09:36:03.535634329 (570h7m3.406717375s) [upload=499830]: Waiting for error group managing segments... Dec 29 09:36:03.808619877 (570h7m3.679702912s) [noise=true , address=145.79.227.39:4014 , piece_id=2T2CTDVIALPHXUQUB243BHOSJNQ6DDPNF3HD7UEGRFHAB4PSHMTA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.813633684 (570h7m3.684716739s) [noise=true , address=145.79.227.14:4838 , piece_id=QBDBP2XZ5PM5QGHOQEHBRAVDMO7XSPSPMM5J2CS2DEY33BME3D5A, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.817536035 (570h7m3.68861908s) [noise=true , address=145.79.227.23:4547 , piece_id=VCQWI5U3AEF25V7T7TW643M3DIE6QWTAHBSFMKX6F7UB3TVQZIZQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.818031269 (570h7m3.689114314s) [noise=true , address=145.79.227.42:4122 , piece_id=33VV7GFCQ4ZYLCJYQ6L3F6YPAZHMNUFKT5YLAGHIOBGZA2IOTYRQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.825472537 (570h7m3.696555582s) [noise=true , address=145.79.227.25:4618 , piece_id=UYCIBF6LE4YCQVVDPR3F7CTX5AKH57RI2EHLY3MNJMJGZ5SGJ4BQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.833374715 (570h7m3.704457771s) [noise=true , address=145.79.227.35:4482 , piece_id=VI4TIIPY6L7BCERHNS4PZVQSFJAGNAUPNQVCY4P767SCBPLYMTYQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.837384349 (570h7m3.708467394s) [noise=true , address=145.79.227.37:4566 , piece_id=4TQ6OYIOF4OJ7V5NBBCVNGHNZDVKM2GVKM2P3V54GOJN4EZ54MYA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.847159914 (570h7m3.718242949s) [noise=true , address=145.79.227.26:4654 , piece_id=ESWH6NQWSGOZ7T42AOWMYZR2GMREMCBOODGOZ6JCBQ6AKZIZQMVQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.861497846 (570h7m3.732580891s) [noise=true , address=145.79.227.43:4157 , piece_id=ED2JLQRCSTPZ36P2NHVCSM2GMYUHFD6BGSVAFJFEZYXSU5LMKIIA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.946243174 (570h7m3.817326219s) [noise=true , address=145.79.227.2:4113 , piece_id=7CGKHRRSMAIHUFTJ6TIDXQJ4SSQD6GX4AXWD2N6CVNH6DR5YXD3A, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.046691734 (570h7m3.917774789s) [noise=true , address=45.91.116.12:4227 , piece_id=2GLFHD4Z7GICBOS32P2BBVKFF5Y25XHVCV6BIIRCQDBINHYVGETA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.071048563 (570h7m3.942131608s) [noise=true , address=77.90.31.17:4773 , piece_id=SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.196182488 (570h7m4.067265533s) [noise=true , address=45.91.116.14:4192 , piece_id=ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.196208267 (570h7m4.067291312s) [seg_pos={0 0}, upload=499830]: Segment reached optimal threshold of 13 pieces. Dec 29 09:36:04.196425641 (570h7m4.067508696s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 09:36:04.196434208 (570h7m4.067517243s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196457381 (570h7m4.067540416s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 09:36:04.196465948 (570h7m4.067548983s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196719781 (570h7m4.067802837s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: context canceled; context canceled Dec 29 09:36:04.196729410 (570h7m4.067812455s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196749057 (570h7m4.067832102s) [seg_pos={0 0}, upload=499830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:36:04.196838557 (570h7m4.067921602s) [upload=499830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:36:04.196845570 (570h7m4.067928615s) [upload=499830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:36:04.483402420 (570h7m4.354485455s) [upload=499830]: Done uploading segments... Dec 29 09:36:04.483427849 (570h7m4.354510884s) [upload=499830]: Upload finished. err: <nil> Duration: 27.505737537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 26.430486666s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:37:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 26.430515812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....XX....X.............X.......X...........................] duration: 3.568817755s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:37:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.568842342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X...X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X..] duration: 5.601728547s trace: https://jaeger.dev.storj.io/trace/5a2746f539f4d260 FAIL (2025-12-29 09:37:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ) context deadline exceeded ======LOGS====== Dec 29 09:36:01.579542678 (570h7m1.450625713s) [upload=499830]: Starting upload Dec 29 09:36:01.579633091 (570h7m1.450716136s) [upload=499830]: Done starting upload Dec 29 09:36:01.579811300 (570h7m1.450894345s) [upload=499830]: Uploading segments... Dec 29 09:36:01.579940636 (570h7m1.451023671s) [upload=499830]: Got next segment. Inline: false Dec 29 09:36:01.579990982 (570h7m1.451074047s) [upload=499830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:36:01.579998797 (570h7m1.451081852s) [upload=499830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:36:01.824871323 (570h7m1.695954388s) [seg_pos={0 0}, upload=499830]: Begin upload segment... Dec 29 09:36:01.824922631 (570h7m1.696005676s) [seg_pos={0 0}, upload=499830]: Done begin upload segment. Dec 29 09:36:01.824956285 (570h7m1.69603933s) [noise=true , address=77.90.31.17:4773 , piece_id=SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.825004567 (570h7m1.696087612s) [noise=true , address=145.79.227.42:4122 , piece_id=33VV7GFCQ4ZYLCJYQ6L3F6YPAZHMNUFKT5YLAGHIOBGZA2IOTYRQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.825568470 (570h7m1.696651506s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826057942 (570h7m1.697140998s) [noise=true , address=45.91.116.12:4227 , piece_id=2GLFHD4Z7GICBOS32P2BBVKFF5Y25XHVCV6BIIRCQDBINHYVGETA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826857184 (570h7m1.697940229s) [noise=true , address=145.79.227.43:4157 , piece_id=ED2JLQRCSTPZ36P2NHVCSM2GMYUHFD6BGSVAFJFEZYXSU5LMKIIA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826891199 (570h7m1.697974244s) [noise=true , address=45.91.116.14:4192 , piece_id=ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827342569 (570h7m1.698425614s) [noise=true , address=145.79.227.2:4113 , piece_id=7CGKHRRSMAIHUFTJ6TIDXQJ4SSQD6GX4AXWD2N6CVNH6DR5YXD3A, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827586253 (570h7m1.698669298s) [noise=true , address=145.79.227.37:4566 , piece_id=4TQ6OYIOF4OJ7V5NBBCVNGHNZDVKM2GVKM2P3V54GOJN4EZ54MYA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827944686 (570h7m1.699027721s) [noise=true , address=145.79.227.23:4547 , piece_id=VCQWI5U3AEF25V7T7TW643M3DIE6QWTAHBSFMKX6F7UB3TVQZIZQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.828307577 (570h7m1.699390632s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.828592890 (570h7m1.699675935s) [noise=true , address=145.79.227.26:4654 , piece_id=ESWH6NQWSGOZ7T42AOWMYZR2GMREMCBOODGOZ6JCBQ6AKZIZQMVQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.829222158 (570h7m1.70033492s) [noise=true , address=145.79.227.14:4838 , piece_id=QBDBP2XZ5PM5QGHOQEHBRAVDMO7XSPSPMM5J2CS2DEY33BME3D5A, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.829961175 (570h7m1.70104422s) [noise=true , address=145.79.227.25:4618 , piece_id=UYCIBF6LE4YCQVVDPR3F7CTX5AKH57RI2EHLY3MNJMJGZ5SGJ4BQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.831771393 (570h7m1.702854438s) [noise=true , address=145.79.227.39:4014 , piece_id=2T2CTDVIALPHXUQUB243BHOSJNQ6DDPNF3HD7UEGRFHAB4PSHMTA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.833759138 (570h7m1.704842173s) [noise=true , address=145.79.227.35:4482 , piece_id=VI4TIIPY6L7BCERHNS4PZVQSFJAGNAUPNQVCY4P767SCBPLYMTYQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.834168076 (570h7m1.705251111s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:03.535615254 (570h7m3.406698309s) [upload=499830]: Next returned nil segment Dec 29 09:36:03.535634329 (570h7m3.406717375s) [upload=499830]: Waiting for error group managing segments... Dec 29 09:36:03.808619877 (570h7m3.679702912s) [noise=true , address=145.79.227.39:4014 , piece_id=2T2CTDVIALPHXUQUB243BHOSJNQ6DDPNF3HD7UEGRFHAB4PSHMTA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.813633684 (570h7m3.684716739s) [noise=true , address=145.79.227.14:4838 , piece_id=QBDBP2XZ5PM5QGHOQEHBRAVDMO7XSPSPMM5J2CS2DEY33BME3D5A, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.817536035 (570h7m3.68861908s) [noise=true , address=145.79.227.23:4547 , piece_id=VCQWI5U3AEF25V7T7TW643M3DIE6QWTAHBSFMKX6F7UB3TVQZIZQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.818031269 (570h7m3.689114314s) [noise=true , address=145.79.227.42:4122 , piece_id=33VV7GFCQ4ZYLCJYQ6L3F6YPAZHMNUFKT5YLAGHIOBGZA2IOTYRQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.825472537 (570h7m3.696555582s) [noise=true , address=145.79.227.25:4618 , piece_id=UYCIBF6LE4YCQVVDPR3F7CTX5AKH57RI2EHLY3MNJMJGZ5SGJ4BQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.833374715 (570h7m3.704457771s) [noise=true , address=145.79.227.35:4482 , piece_id=VI4TIIPY6L7BCERHNS4PZVQSFJAGNAUPNQVCY4P767SCBPLYMTYQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.837384349 (570h7m3.708467394s) [noise=true , address=145.79.227.37:4566 , piece_id=4TQ6OYIOF4OJ7V5NBBCVNGHNZDVKM2GVKM2P3V54GOJN4EZ54MYA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.847159914 (570h7m3.718242949s) [noise=true , address=145.79.227.26:4654 , piece_id=ESWH6NQWSGOZ7T42AOWMYZR2GMREMCBOODGOZ6JCBQ6AKZIZQMVQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.861497846 (570h7m3.732580891s) [noise=true , address=145.79.227.43:4157 , piece_id=ED2JLQRCSTPZ36P2NHVCSM2GMYUHFD6BGSVAFJFEZYXSU5LMKIIA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.946243174 (570h7m3.817326219s) [noise=true , address=145.79.227.2:4113 , piece_id=7CGKHRRSMAIHUFTJ6TIDXQJ4SSQD6GX4AXWD2N6CVNH6DR5YXD3A, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.046691734 (570h7m3.917774789s) [noise=true , address=45.91.116.12:4227 , piece_id=2GLFHD4Z7GICBOS32P2BBVKFF5Y25XHVCV6BIIRCQDBINHYVGETA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.071048563 (570h7m3.942131608s) [noise=true , address=77.90.31.17:4773 , piece_id=SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.196182488 (570h7m4.067265533s) [noise=true , address=45.91.116.14:4192 , piece_id=ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.196208267 (570h7m4.067291312s) [seg_pos={0 0}, upload=499830]: Segment reached optimal threshold of 13 pieces. Dec 29 09:36:04.196425641 (570h7m4.067508696s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 09:36:04.196434208 (570h7m4.067517243s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196457381 (570h7m4.067540416s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 09:36:04.196465948 (570h7m4.067548983s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196719781 (570h7m4.067802837s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: context canceled; context canceled Dec 29 09:36:04.196729410 (570h7m4.067812455s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196749057 (570h7m4.067832102s) [seg_pos={0 0}, upload=499830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:36:04.196838557 (570h7m4.067921602s) [upload=499830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:36:04.196845570 (570h7m4.067928615s) [upload=499830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:36:04.483402420 (570h7m4.354485455s) [upload=499830]: Done uploading segments... Dec 29 09:36:04.483427849 (570h7m4.354510884s) [upload=499830]: Upload finished. err: <nil> Duration: 5.601757673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 41.1681515s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:37:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 41.168168383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....XX....X.............X.......X..........................] duration: 3.40185124s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:37:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.401879683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.X...X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X.] duration: 6.280868969s trace: https://jaeger.dev.storj.io/trace/5a2746f539f4d260 FAIL (2025-12-29 09:36:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ) context deadline exceeded ======LOGS====== Dec 29 09:36:01.579542678 (570h7m1.450625713s) [upload=499830]: Starting upload Dec 29 09:36:01.579633091 (570h7m1.450716136s) [upload=499830]: Done starting upload Dec 29 09:36:01.579811300 (570h7m1.450894345s) [upload=499830]: Uploading segments... Dec 29 09:36:01.579940636 (570h7m1.451023671s) [upload=499830]: Got next segment. Inline: false Dec 29 09:36:01.579990982 (570h7m1.451074047s) [upload=499830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:36:01.579998797 (570h7m1.451081852s) [upload=499830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:36:01.824871323 (570h7m1.695954388s) [seg_pos={0 0}, upload=499830]: Begin upload segment... Dec 29 09:36:01.824922631 (570h7m1.696005676s) [seg_pos={0 0}, upload=499830]: Done begin upload segment. Dec 29 09:36:01.824956285 (570h7m1.69603933s) [noise=true , address=77.90.31.17:4773 , piece_id=SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.825004567 (570h7m1.696087612s) [noise=true , address=145.79.227.42:4122 , piece_id=33VV7GFCQ4ZYLCJYQ6L3F6YPAZHMNUFKT5YLAGHIOBGZA2IOTYRQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.825568470 (570h7m1.696651506s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826057942 (570h7m1.697140998s) [noise=true , address=45.91.116.12:4227 , piece_id=2GLFHD4Z7GICBOS32P2BBVKFF5Y25XHVCV6BIIRCQDBINHYVGETA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826857184 (570h7m1.697940229s) [noise=true , address=145.79.227.43:4157 , piece_id=ED2JLQRCSTPZ36P2NHVCSM2GMYUHFD6BGSVAFJFEZYXSU5LMKIIA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826891199 (570h7m1.697974244s) [noise=true , address=45.91.116.14:4192 , piece_id=ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827342569 (570h7m1.698425614s) [noise=true , address=145.79.227.2:4113 , piece_id=7CGKHRRSMAIHUFTJ6TIDXQJ4SSQD6GX4AXWD2N6CVNH6DR5YXD3A, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827586253 (570h7m1.698669298s) [noise=true , address=145.79.227.37:4566 , piece_id=4TQ6OYIOF4OJ7V5NBBCVNGHNZDVKM2GVKM2P3V54GOJN4EZ54MYA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827944686 (570h7m1.699027721s) [noise=true , address=145.79.227.23:4547 , piece_id=VCQWI5U3AEF25V7T7TW643M3DIE6QWTAHBSFMKX6F7UB3TVQZIZQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.828307577 (570h7m1.699390632s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.828592890 (570h7m1.699675935s) [noise=true , address=145.79.227.26:4654 , piece_id=ESWH6NQWSGOZ7T42AOWMYZR2GMREMCBOODGOZ6JCBQ6AKZIZQMVQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.829222158 (570h7m1.70033492s) [noise=true , address=145.79.227.14:4838 , piece_id=QBDBP2XZ5PM5QGHOQEHBRAVDMO7XSPSPMM5J2CS2DEY33BME3D5A, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.829961175 (570h7m1.70104422s) [noise=true , address=145.79.227.25:4618 , piece_id=UYCIBF6LE4YCQVVDPR3F7CTX5AKH57RI2EHLY3MNJMJGZ5SGJ4BQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.831771393 (570h7m1.702854438s) [noise=true , address=145.79.227.39:4014 , piece_id=2T2CTDVIALPHXUQUB243BHOSJNQ6DDPNF3HD7UEGRFHAB4PSHMTA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.833759138 (570h7m1.704842173s) [noise=true , address=145.79.227.35:4482 , piece_id=VI4TIIPY6L7BCERHNS4PZVQSFJAGNAUPNQVCY4P767SCBPLYMTYQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.834168076 (570h7m1.705251111s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:03.535615254 (570h7m3.406698309s) [upload=499830]: Next returned nil segment Dec 29 09:36:03.535634329 (570h7m3.406717375s) [upload=499830]: Waiting for error group managing segments... Dec 29 09:36:03.808619877 (570h7m3.679702912s) [noise=true , address=145.79.227.39:4014 , piece_id=2T2CTDVIALPHXUQUB243BHOSJNQ6DDPNF3HD7UEGRFHAB4PSHMTA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.813633684 (570h7m3.684716739s) [noise=true , address=145.79.227.14:4838 , piece_id=QBDBP2XZ5PM5QGHOQEHBRAVDMO7XSPSPMM5J2CS2DEY33BME3D5A, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.817536035 (570h7m3.68861908s) [noise=true , address=145.79.227.23:4547 , piece_id=VCQWI5U3AEF25V7T7TW643M3DIE6QWTAHBSFMKX6F7UB3TVQZIZQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.818031269 (570h7m3.689114314s) [noise=true , address=145.79.227.42:4122 , piece_id=33VV7GFCQ4ZYLCJYQ6L3F6YPAZHMNUFKT5YLAGHIOBGZA2IOTYRQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.825472537 (570h7m3.696555582s) [noise=true , address=145.79.227.25:4618 , piece_id=UYCIBF6LE4YCQVVDPR3F7CTX5AKH57RI2EHLY3MNJMJGZ5SGJ4BQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.833374715 (570h7m3.704457771s) [noise=true , address=145.79.227.35:4482 , piece_id=VI4TIIPY6L7BCERHNS4PZVQSFJAGNAUPNQVCY4P767SCBPLYMTYQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.837384349 (570h7m3.708467394s) [noise=true , address=145.79.227.37:4566 , piece_id=4TQ6OYIOF4OJ7V5NBBCVNGHNZDVKM2GVKM2P3V54GOJN4EZ54MYA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.847159914 (570h7m3.718242949s) [noise=true , address=145.79.227.26:4654 , piece_id=ESWH6NQWSGOZ7T42AOWMYZR2GMREMCBOODGOZ6JCBQ6AKZIZQMVQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.861497846 (570h7m3.732580891s) [noise=true , address=145.79.227.43:4157 , piece_id=ED2JLQRCSTPZ36P2NHVCSM2GMYUHFD6BGSVAFJFEZYXSU5LMKIIA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.946243174 (570h7m3.817326219s) [noise=true , address=145.79.227.2:4113 , piece_id=7CGKHRRSMAIHUFTJ6TIDXQJ4SSQD6GX4AXWD2N6CVNH6DR5YXD3A, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.046691734 (570h7m3.917774789s) [noise=true , address=45.91.116.12:4227 , piece_id=2GLFHD4Z7GICBOS32P2BBVKFF5Y25XHVCV6BIIRCQDBINHYVGETA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.071048563 (570h7m3.942131608s) [noise=true , address=77.90.31.17:4773 , piece_id=SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.196182488 (570h7m4.067265533s) [noise=true , address=45.91.116.14:4192 , piece_id=ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.196208267 (570h7m4.067291312s) [seg_pos={0 0}, upload=499830]: Segment reached optimal threshold of 13 pieces. Dec 29 09:36:04.196425641 (570h7m4.067508696s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 09:36:04.196434208 (570h7m4.067517243s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196457381 (570h7m4.067540416s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 09:36:04.196465948 (570h7m4.067548983s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196719781 (570h7m4.067802837s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: context canceled; context canceled Dec 29 09:36:04.196729410 (570h7m4.067812455s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196749057 (570h7m4.067832102s) [seg_pos={0 0}, upload=499830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:36:04.196838557 (570h7m4.067921602s) [upload=499830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:36:04.196845570 (570h7m4.067928615s) [upload=499830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:36:04.483402420 (570h7m4.354485455s) [upload=499830]: Done uploading segments... Dec 29 09:36:04.483427849 (570h7m4.354510884s) [upload=499830]: Upload finished. err: <nil> Duration: 6.280900158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.X...X...XX.X.XX..XXXX..X.....XX..XX....X.XX...XX...X.X.X] duration: 30.002453651s trace: https://jaeger.dev.storj.io/trace/5a2746f539f4d260 FAIL (2025-12-29 09:36:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ) context deadline exceeded ======LOGS====== Dec 29 09:36:01.579542678 (570h7m1.450625713s) [upload=499830]: Starting upload Dec 29 09:36:01.579633091 (570h7m1.450716136s) [upload=499830]: Done starting upload Dec 29 09:36:01.579811300 (570h7m1.450894345s) [upload=499830]: Uploading segments... Dec 29 09:36:01.579940636 (570h7m1.451023671s) [upload=499830]: Got next segment. Inline: false Dec 29 09:36:01.579990982 (570h7m1.451074047s) [upload=499830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:36:01.579998797 (570h7m1.451081852s) [upload=499830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:36:01.824871323 (570h7m1.695954388s) [seg_pos={0 0}, upload=499830]: Begin upload segment... Dec 29 09:36:01.824922631 (570h7m1.696005676s) [seg_pos={0 0}, upload=499830]: Done begin upload segment. Dec 29 09:36:01.824956285 (570h7m1.69603933s) [noise=true , address=77.90.31.17:4773 , piece_id=SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.825004567 (570h7m1.696087612s) [noise=true , address=145.79.227.42:4122 , piece_id=33VV7GFCQ4ZYLCJYQ6L3F6YPAZHMNUFKT5YLAGHIOBGZA2IOTYRQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.825568470 (570h7m1.696651506s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826057942 (570h7m1.697140998s) [noise=true , address=45.91.116.12:4227 , piece_id=2GLFHD4Z7GICBOS32P2BBVKFF5Y25XHVCV6BIIRCQDBINHYVGETA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826857184 (570h7m1.697940229s) [noise=true , address=145.79.227.43:4157 , piece_id=ED2JLQRCSTPZ36P2NHVCSM2GMYUHFD6BGSVAFJFEZYXSU5LMKIIA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.826891199 (570h7m1.697974244s) [noise=true , address=45.91.116.14:4192 , piece_id=ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827342569 (570h7m1.698425614s) [noise=true , address=145.79.227.2:4113 , piece_id=7CGKHRRSMAIHUFTJ6TIDXQJ4SSQD6GX4AXWD2N6CVNH6DR5YXD3A, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827586253 (570h7m1.698669298s) [noise=true , address=145.79.227.37:4566 , piece_id=4TQ6OYIOF4OJ7V5NBBCVNGHNZDVKM2GVKM2P3V54GOJN4EZ54MYA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.827944686 (570h7m1.699027721s) [noise=true , address=145.79.227.23:4547 , piece_id=VCQWI5U3AEF25V7T7TW643M3DIE6QWTAHBSFMKX6F7UB3TVQZIZQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.828307577 (570h7m1.699390632s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.828592890 (570h7m1.699675935s) [noise=true , address=145.79.227.26:4654 , piece_id=ESWH6NQWSGOZ7T42AOWMYZR2GMREMCBOODGOZ6JCBQ6AKZIZQMVQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.829222158 (570h7m1.70033492s) [noise=true , address=145.79.227.14:4838 , piece_id=QBDBP2XZ5PM5QGHOQEHBRAVDMO7XSPSPMM5J2CS2DEY33BME3D5A, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.829961175 (570h7m1.70104422s) [noise=true , address=145.79.227.25:4618 , piece_id=UYCIBF6LE4YCQVVDPR3F7CTX5AKH57RI2EHLY3MNJMJGZ5SGJ4BQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.831771393 (570h7m1.702854438s) [noise=true , address=145.79.227.39:4014 , piece_id=2T2CTDVIALPHXUQUB243BHOSJNQ6DDPNF3HD7UEGRFHAB4PSHMTA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.833759138 (570h7m1.704842173s) [noise=true , address=145.79.227.35:4482 , piece_id=VI4TIIPY6L7BCERHNS4PZVQSFJAGNAUPNQVCY4P767SCBPLYMTYQ, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:01.834168076 (570h7m1.705251111s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Uploading piece... Dec 29 09:36:03.535615254 (570h7m3.406698309s) [upload=499830]: Next returned nil segment Dec 29 09:36:03.535634329 (570h7m3.406717375s) [upload=499830]: Waiting for error group managing segments... Dec 29 09:36:03.808619877 (570h7m3.679702912s) [noise=true , address=145.79.227.39:4014 , piece_id=2T2CTDVIALPHXUQUB243BHOSJNQ6DDPNF3HD7UEGRFHAB4PSHMTA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.813633684 (570h7m3.684716739s) [noise=true , address=145.79.227.14:4838 , piece_id=QBDBP2XZ5PM5QGHOQEHBRAVDMO7XSPSPMM5J2CS2DEY33BME3D5A, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.817536035 (570h7m3.68861908s) [noise=true , address=145.79.227.23:4547 , piece_id=VCQWI5U3AEF25V7T7TW643M3DIE6QWTAHBSFMKX6F7UB3TVQZIZQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.818031269 (570h7m3.689114314s) [noise=true , address=145.79.227.42:4122 , piece_id=33VV7GFCQ4ZYLCJYQ6L3F6YPAZHMNUFKT5YLAGHIOBGZA2IOTYRQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.825472537 (570h7m3.696555582s) [noise=true , address=145.79.227.25:4618 , piece_id=UYCIBF6LE4YCQVVDPR3F7CTX5AKH57RI2EHLY3MNJMJGZ5SGJ4BQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.833374715 (570h7m3.704457771s) [noise=true , address=145.79.227.35:4482 , piece_id=VI4TIIPY6L7BCERHNS4PZVQSFJAGNAUPNQVCY4P767SCBPLYMTYQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.837384349 (570h7m3.708467394s) [noise=true , address=145.79.227.37:4566 , piece_id=4TQ6OYIOF4OJ7V5NBBCVNGHNZDVKM2GVKM2P3V54GOJN4EZ54MYA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.847159914 (570h7m3.718242949s) [noise=true , address=145.79.227.26:4654 , piece_id=ESWH6NQWSGOZ7T42AOWMYZR2GMREMCBOODGOZ6JCBQ6AKZIZQMVQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.861497846 (570h7m3.732580891s) [noise=true , address=145.79.227.43:4157 , piece_id=ED2JLQRCSTPZ36P2NHVCSM2GMYUHFD6BGSVAFJFEZYXSU5LMKIIA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:03.946243174 (570h7m3.817326219s) [noise=true , address=145.79.227.2:4113 , piece_id=7CGKHRRSMAIHUFTJ6TIDXQJ4SSQD6GX4AXWD2N6CVNH6DR5YXD3A, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.046691734 (570h7m3.917774789s) [noise=true , address=45.91.116.12:4227 , piece_id=2GLFHD4Z7GICBOS32P2BBVKFF5Y25XHVCV6BIIRCQDBINHYVGETA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.071048563 (570h7m3.942131608s) [noise=true , address=77.90.31.17:4773 , piece_id=SPKTY5V753A3C7AIWJVHU2KUPK5MFTQRIUA5HSCTAHUA5YT3RPAA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.196182488 (570h7m4.067265533s) [noise=true , address=45.91.116.14:4192 , piece_id=ECNCLRAPRMT2AUBPFDDDAISQLD7H7VBNMNKBRSASLGIMN5YYRVYQ, seg_pos={0 0}, upload=499830]: Done uploading piece. err: <nil> Dec 29 09:36:04.196208267 (570h7m4.067291312s) [seg_pos={0 0}, upload=499830]: Segment reached optimal threshold of 13 pieces. Dec 29 09:36:04.196425641 (570h7m4.067508696s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 09:36:04.196434208 (570h7m4.067517243s) [noise=true , address=145.79.227.44:4194 , piece_id=UQQ7K37W7VJWA3ZKP6ZVZT56AFPBDG2C3ZR27OGGT3U6TPG5HQFA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196457381 (570h7m4.067540416s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 09:36:04.196465948 (570h7m4.067548983s) [noise=true , address=77.90.31.12:4590 , piece_id=S4D7QY6RYLUXOPXA5NTVBDXVQ4DIW3TWMWLJEGT7GFKNO5KAFCXA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196719781 (570h7m4.067802837s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: context canceled; context canceled Dec 29 09:36:04.196729410 (570h7m4.067812455s) [noise=true , address=45.91.116.13:4156 , piece_id=5L23V2Q32WJ5XXM7VBJPITKT243QANVHWGU7MTBZDGLEMJXFHMYA, seg_pos={0 0}, upload=499830]: Long tail cancellation due to optimal threshold Dec 29 09:36:04.196749057 (570h7m4.067832102s) [seg_pos={0 0}, upload=499830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:36:04.196838557 (570h7m4.067921602s) [upload=499830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:36:04.196845570 (570h7m4.067928615s) [upload=499830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:36:04.483402420 (570h7m4.354485455s) [upload=499830]: Done uploading segments... Dec 29 09:36:04.483427849 (570h7m4.354510884s) [upload=499830]: Upload finished. err: <nil> Duration: 30.002467646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 36.652598989s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:36:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 36.652615149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......XX....X.............X.......X.........................] duration: 3.536240977s trace: https://jaeger.dev.storj.io/trace/d6a1bb0561284e6 FAIL (2025-12-29 09:35:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.536267488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 44.634281639s trace: https://jaeger.dev.storj.io/trace/635cd8bc3650ee07 FAIL (2025-12-29 09:35:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d=499744]: Uploading piece... Dec 29 09:29:58.279548806 (570h0m58.150631841s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.279622667 (570h0m58.150705702s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280280058 (570h0m58.151363123s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280403724 (570h0m58.151486789s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.280813725 (570h0m58.15189676s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281083499 (570h0m58.152166534s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.281329607 (570h0m58.152412642s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282173785 (570h0m58.15325682s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.282539831 (570h0m58.153622876s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283048400 (570h0m58.154131445s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283442660 (570h0m58.154525705s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.283838394 (570h0m58.15492143s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.284255108 (570h0m58.155338143s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288374702 (570h0m58.159457747s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.288692527 (570h0m58.159775572s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:58.290603546 (570h0m58.161686611s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Uploading piece... Dec 29 09:29:59.609369428 (570h0m59.480452463s) [upload=499744]: Next returned nil segment Dec 29 09:29:59.609400337 (570h0m59.480483382s) [upload=499744]: Waiting for error group managing segments... Dec 29 09:29:59.944102013 (570h0m59.815185048s) [noise=true , address=64.49.13.26:4946 , piece_id=IO4OL4OMTJKNA6PTGLLUMAJIM22U2DMQDDXXR4A3GNQ72SGFOS7A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.152043536 (570h1m0.023126571s) [noise=true , address=145.79.227.29:4958 , piece_id=REQUMMDJBDBOUY44XRNYROCBLWNUEPMEGONF4HMBGIGYJ3WTC5XQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.157412580 (570h1m0.028495616s) [noise=true , address=145.79.227.41:4100 , piece_id=Q6V3ADRVJX6G4SGLUGATYTDYR6NFY2Q7HRVMSB7QZXI2CY3B4VOA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.161488812 (570h1m0.032571847s) [noise=true , address=145.79.227.46:4632 , piece_id=W2AGQZ4HF6S7KMVM4MFY3BULOF42WV7OJVJ6774KYURE6MHJ46LQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.343626932 (570h1m0.214709977s) [noise=true , address=77.90.31.21:4910 , piece_id=WRETVKKIXK66RVN6QBIC7J7ZYFVL7TBYVHS5TD3K4GPUBY5YEU4Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.422875725 (570h1m0.29395877s) [noise=true , address=77.90.31.11:4585 , piece_id=YHXDK3RDR4OPNLCJKRKEPS2PYXEAZVN7EAWLSAGA2ISFZ47US5QA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.514065344 (570h1m0.385148389s) [noise=true , address=77.90.31.18:4819 , piece_id=QDZUZATJRZDV4475EWW3WSVRCROL7DDNLSPHCTCM4IBIFQRRYDNA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.623679628 (570h1m0.494762663s) [noise=true , address=145.79.227.12:4771 , piece_id=YHJGLC3KCC4WMKWCY5P6GAQQRFNZUW3RMSLSHEOFEOEZNS7FDDXA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:00.957178323 (570h1m0.828261358s) [noise=true , address=145.79.227.35:4507 , piece_id=OOX2ZJCNNMG3YNYVXTLDF4QCQAGYWMPZXPFCDNLP7D2JMN6DUQLA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.026259996 (570h1m0.897343041s) [noise=true , address=145.79.227.44:4202 , piece_id=IAAKQHRXJAPWTNHUNDACXG4VTS6KUQ7RDTO7JSILXN5PIJIAPLHA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.044865523 (570h1m0.915948569s) [noise=true , address=145.79.227.40:4083 , piece_id=3FK6HKGQIA2UZFIHZH5QMWG5ZZYTJCS4I7JYUTD2PXTNBSAAS53A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.056819856 (570h1m0.927902891s) [noise=true , address=145.79.227.49:4344 , piece_id=A3O22E7WMQUFZ6LU2BT4QK53UAKA66HF53WVCT4QF2XWLMOAJ75A, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.116784799 (570h1m0.987867844s) [noise=true , address=77.90.31.12:4611 , piece_id=3FWD6FD3BQQYHEKQHTXJKPOO3L4JPL5MFGSWJAQZCLW2R7E2UTRQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125873055 (570h1m0.9969561s) [noise=true , address=77.90.31.17:4802 , piece_id=7KUAZLXISFFJ27TSYXUVMW223QYZ77MF6QHSQIK7WNFUNWGEQRWA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: <nil> Dec 29 09:30:01.125899175 (570h1m0.99698222s) [seg_pos={0 0}, upload=499744]: Segment reached optimal threshold of 14 pieces. Dec 29 09:30:01.126201952 (570h1m0.997284997s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60282->64.49.13.169:4707: use of closed network connection Dec 29 09:30:01.126210538 (570h1m0.997293583s) [noise=true , address=64.49.13.169:4707 , piece_id=RUO373SATXCFFLXYAMPWP77EPQNBZQY5AKHQTC63OZMQTK3JZSDQ, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126261885 (570h1m0.99734493s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126272987 (570h1m0.997356032s) [noise=true , address=145.79.227.33:4875 , piece_id=EL2GWOHFDPK6U4HXZ7JA6DDCZZU3YEKBJFSKLD2U2Q3ULXGEQQBA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126470963 (570h1m0.997554008s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126479650 (570h1m0.997562685s) [noise=true , address=193.149.164.143:4701 , piece_id=Y5C6ZM5FHKHSYUCWJVRKHD3KIKEJ7ZF2SSMOQY3R3UH5DIWJJJ3Q, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126563530 (570h1m0.997646575s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 29 09:30:01.126578748 (570h1m0.997661793s) [noise=true , address=145.79.227.34:4929 , piece_id=VMAWAZNRIT3MBFWT7ZOEPIIJ3URP3ZNEVRDCL7NWSPTVPMZB7BAA, seg_pos={0 0}, upload=499744]: Long tail cancellation due to optimal threshold Dec 29 09:30:01.126654834 (570h1m0.997737879s) [seg_pos={0 0}, upload=499744]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:30:01.126790652 (570h1m0.997873687s) [upload=499744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:30:01.126797445 (570h1m0.99788049s) [upload=499744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:30:01.417532391 (570h1m1.288615436s) [upload=499744]: Done uploading segments... Dec 29 09:30:01.417560435 (570h1m1.28864348s) [upload=499744]: Upload finished. err: <nil> Duration: 44.634317057s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)