Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 895374 reports (Page 11397 of 17908)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X.....] duration: 5.814766591s trace: https://jaeger.dev.storj.io/trace/69d78e97cabd159d FAIL (2025-12-26 15:02:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ) context deadline exceeded ======LOGS====== Dec 26 14:58:32.374358736 (503h29m32.245441811s) [upload=441835]: Starting upload Dec 26 14:58:32.374448627 (503h29m32.245531693s) [upload=441835]: Done starting upload Dec 26 14:58:32.374517879 (503h29m32.245600914s) [upload=441835]: Uploading segments... Dec 26 14:58:32.374719773 (503h29m32.245802818s) [upload=441835]: Got next segment. Inline: false Dec 26 14:58:32.374776652 (503h29m32.245859697s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:58:32.374783274 (503h29m32.245866309s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:58:32.623839040 (503h29m32.494922085s) [seg_pos={0 0}, upload=441835]: Begin upload segment... Dec 26 14:58:32.623900226 (503h29m32.494983262s) [seg_pos={0 0}, upload=441835]: Done begin upload segment. Dec 26 14:58:32.623952255 (503h29m32.4950353s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.623959939 (503h29m32.495042975s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624027368 (503h29m32.495110413s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624660373 (503h29m32.495743408s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625232914 (503h29m32.496315959s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625914471 (503h29m32.496997516s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.626458728 (503h29m32.497541773s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627053380 (503h29m32.498136415s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627081644 (503h29m32.498164689s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627849155 (503h29m32.498932191s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628613802 (503h29m32.499696847s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628801959 (503h29m32.499885004s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.629437300 (503h29m32.500520335s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630042502 (503h29m32.501125547s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630237995 (503h29m32.50132104s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630844299 (503h29m32.501927354s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:34.333752309 (503h29m34.204835355s) [upload=441835]: Next returned nil segment Dec 26 14:58:34.333780083 (503h29m34.204863118s) [upload=441835]: Waiting for error group managing segments... Dec 26 14:58:34.629882783 (503h29m34.500965818s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.630250563 (503h29m34.501333598s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.635559923 (503h29m34.506642968s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.637515627 (503h29m34.508598672s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645708077 (503h29m34.516791112s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645931903 (503h29m34.517014948s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.646571351 (503h29m34.517654386s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.652497244 (503h29m34.523580279s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.653752085 (503h29m34.52483513s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.657867030 (503h29m34.528950065s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.664311101 (503h29m34.535394146s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.933887380 (503h29m34.804970415s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950485897 (503h29m34.821568932s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950504161 (503h29m34.821587196s) [seg_pos={0 0}, upload=441835]: Segment reached optimal threshold of 13 pieces. Dec 26 14:58:34.950668044 (503h29m34.821751079s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950673263 (503h29m34.821756298s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950767593 (503h29m34.821850628s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950777642 (503h29m34.821860687s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950845972 (503h29m34.821929007s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950852044 (503h29m34.821935089s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950867774 (503h29m34.821950809s) [seg_pos={0 0}, upload=441835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:58:34.950938940 (503h29m34.822021975s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:58:34.950943258 (503h29m34.822026293s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:58:35.245768465 (503h29m35.116851521s) [upload=441835]: Done uploading segments... Dec 26 14:58:35.245792401 (503h29m35.116875446s) [upload=441835]: Upload finished. err: <nil> Duration: 5.814792079s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 8.118832809s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 15:01:56 UTC) orbiter warning: context deadline exceeded Duration: 8.118859259s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.X] duration: 5.001341907s trace: https://jaeger.dev.storj.io/trace/bcbb8eeb1a95059 FAIL (2025-12-26 15:01:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001366044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXXX] duration: 1m0.002323451s trace: https://jaeger.dev.storj.io/trace/7942882354f43b3d FAIL (2025-12-26 15:01:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== UJNWCDKE4CZIQYJZT6YRKONGXSQ, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.209105447 (503h31m28.080188492s) [noise=true , address=77.90.31.18:4818 , piece_id=VOIUNZ4CXSLMD53TKNK2GZAOIO7W5BUW2MNPYDRYL7MN5GU5DPSA, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.209280942 (503h31m28.080363987s) [noise=true , address=145.79.227.30:4787 , piece_id=TCFQCB22TIWKEP4GIKAUS6JOCYHMK6WGRIVYQ7NNS4L5SMPYGUFQ, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.209706031 (503h31m28.080789066s) [noise=true , address=64.49.13.140:4509 , piece_id=RU23ZDVPBNALPQEJH725YX45VZ7JMEMPEOQERBGHDLIA26SIARGA, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.209729085 (503h31m28.08081213s) [noise=true , address=77.90.31.15:4718 , piece_id=Y42TV5MG77LQEU7TLJNVBHJLVQ4U6KU5XJMFMI5DQE2BYCBSBCGQ, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.210059134 (503h31m28.081142169s) [noise=true , address=64.49.13.139:4474 , piece_id=DQRCIHV32UFS53HJ6TTFQSFBUTZANCWOYETHYTDVWWWWXVQ4XOMQ, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.210121192 (503h31m28.081204227s) [noise=true , address=145.79.227.52:4460 , piece_id=SNWW37ACJRSQ4YCVKHAQY5AJWHFBRCYCMK2P5T2B6AWXQOG7AU6A, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.210373803 (503h31m28.081456848s) [noise=true , address=145.79.227.7:4298 , piece_id=NEN367YU35G5J3GUNX4FNKDV23UD2WUMTWX3OCJJ54FKVLNLIZVA, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.210529980 (503h31m28.081613015s) [noise=true , address=64.49.13.169:4713 , piece_id=KVYCBAMSYC4Y6RSHFXKXBCVMJVBTFATWHU3DC66HVS5RCOG4CQJQ, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.210726514 (503h31m28.081809559s) [noise=true , address=145.79.227.5:4237 , piece_id=ZDNTRDTPBH3J5UA5OGL4O4HVEVWOIZOI5NTT77UT5ZGIBXOH45LQ, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.210928088 (503h31m28.082011133s) [noise=true , address=145.79.227.33:4895 , piece_id=EMDPFDFIMRURLTLSVTDYFIHI7NH2BICXZAPUSC5HR57B7APQ5AFA, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.211147937 (503h31m28.082230982s) [noise=true , address=77.90.31.10:4530 , piece_id=7DRY6OTDLT27YW5KPH55A2S62JXF7QHJLUPQE5CN5PV5OEH6NGKA, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.211453549 (503h31m28.082536584s) [noise=true , address=145.79.227.34:4918 , piece_id=HBAAT6NJCTWKY3L2NI4PQQWCEMD365KZWWQFWDGJI3ZXZQ2ZVIVA, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.211518883 (503h31m28.082601928s) [noise=true , address=77.90.31.11:4580 , piece_id=BB3Q6CFZ4HFFHWWWMPU4K76T4XYNIPTTUZQK6X7QVFUW2XHI4SPQ, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.211832250 (503h31m28.082915285s) [noise=true , address=77.90.31.13:4649 , piece_id=U7QWW44Q23IMCVNZEAMXYOHDP4XZFCGCAJ76A7YOTMHQYYHNPU2A, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:28.211895531 (503h31m28.082978566s) [noise=true , address=145.79.227.40:4062 , piece_id=RV2OPNYHTJEENSDDSKPVUGBUJDKO3JLK7HNY4QHUIQC63HYO62YQ, seg_pos={0 0}, upload=441864]: Uploading piece... Dec 26 15:00:29.668436126 (503h31m29.539519181s) [upload=441864]: Next returned nil segment Dec 26 15:00:29.668451224 (503h31m29.539534269s) [upload=441864]: Waiting for error group managing segments... Dec 26 15:00:30.012265777 (503h31m29.883348822s) [noise=true , address=64.49.13.140:4509 , piece_id=RU23ZDVPBNALPQEJH725YX45VZ7JMEMPEOQERBGHDLIA26SIARGA, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.080704135 (503h31m29.95178718s) [noise=true , address=145.79.227.34:4918 , piece_id=HBAAT6NJCTWKY3L2NI4PQQWCEMD365KZWWQFWDGJI3ZXZQ2ZVIVA, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.089009380 (503h31m29.960092415s) [noise=true , address=145.79.227.7:4298 , piece_id=NEN367YU35G5J3GUNX4FNKDV23UD2WUMTWX3OCJJ54FKVLNLIZVA, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.089999094 (503h31m29.961082139s) [noise=true , address=145.79.227.52:4460 , piece_id=SNWW37ACJRSQ4YCVKHAQY5AJWHFBRCYCMK2P5T2B6AWXQOG7AU6A, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.107890392 (503h31m29.978973437s) [noise=true , address=145.79.227.40:4062 , piece_id=RV2OPNYHTJEENSDDSKPVUGBUJDKO3JLK7HNY4QHUIQC63HYO62YQ, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.119851688 (503h31m29.990934733s) [noise=true , address=145.79.227.5:4237 , piece_id=ZDNTRDTPBH3J5UA5OGL4O4HVEVWOIZOI5NTT77UT5ZGIBXOH45LQ, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.369997680 (503h31m30.241080725s) [noise=true , address=77.90.31.21:4884 , piece_id=QHDXY6L4SA3T2QWH5LHNBKP44AQMDYR7UV6H3PDMRWJY6AL5DPYA, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.374738287 (503h31m30.245821332s) [noise=true , address=77.90.31.18:4818 , piece_id=VOIUNZ4CXSLMD53TKNK2GZAOIO7W5BUW2MNPYDRYL7MN5GU5DPSA, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.396556502 (503h31m30.267639537s) [noise=true , address=77.90.31.13:4649 , piece_id=U7QWW44Q23IMCVNZEAMXYOHDP4XZFCGCAJ76A7YOTMHQYYHNPU2A, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.433585863 (503h31m30.304668898s) [noise=true , address=77.90.31.10:4530 , piece_id=7DRY6OTDLT27YW5KPH55A2S62JXF7QHJLUPQE5CN5PV5OEH6NGKA, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.723183426 (503h31m30.594266471s) [noise=true , address=45.91.116.13:4180 , piece_id=MFDF2ITSVDDUPRZ4RT275O2KSUJNWCDKE4CZIQYJZT6YRKONGXSQ, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:30.917787597 (503h31m30.788870642s) [noise=true , address=64.49.13.139:4474 , piece_id=DQRCIHV32UFS53HJ6TTFQSFBUTZANCWOYETHYTDVWWWWXVQ4XOMQ, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:31.022392962 (503h31m30.893476007s) [noise=true , address=145.79.227.44:4218 , piece_id=DFRAUSG6KFWXBVJ5ARCSMABGG3YJAXF4ARMYNLGKJ7MCPVTC3ZWA, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:31.074335320 (503h31m30.945418366s) [noise=true , address=145.79.227.33:4895 , piece_id=EMDPFDFIMRURLTLSVTDYFIHI7NH2BICXZAPUSC5HR57B7APQ5AFA, seg_pos={0 0}, upload=441864]: Done uploading piece. err: <nil> Dec 26 15:00:31.074362191 (503h31m30.945445236s) [seg_pos={0 0}, upload=441864]: Segment reached optimal threshold of 14 pieces. Dec 26 15:00:31.074775638 (503h31m30.945858683s) [noise=true , address=64.49.13.169:4713 , piece_id=KVYCBAMSYC4Y6RSHFXKXBCVMJVBTFATWHU3DC66HVS5RCOG4CQJQ, seg_pos={0 0}, upload=441864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:54568->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:54568->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54568->64.49.13.169:4713: use of closed network connection Dec 26 15:00:31.074796929 (503h31m30.945879974s) [noise=true , address=64.49.13.169:4713 , piece_id=KVYCBAMSYC4Y6RSHFXKXBCVMJVBTFATWHU3DC66HVS5RCOG4CQJQ, seg_pos={0 0}, upload=441864]: Long tail cancellation due to optimal threshold Dec 26 15:00:31.074871671 (503h31m30.945954706s) [noise=true , address=77.90.31.15:4718 , piece_id=Y42TV5MG77LQEU7TLJNVBHJLVQ4U6KU5XJMFMI5DQE2BYCBSBCGQ, seg_pos={0 0}, upload=441864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: expected piece hash; context canceled; EOF Dec 26 15:00:31.074882201 (503h31m30.945965246s) [noise=true , address=77.90.31.15:4718 , piece_id=Y42TV5MG77LQEU7TLJNVBHJLVQ4U6KU5XJMFMI5DQE2BYCBSBCGQ, seg_pos={0 0}, upload=441864]: Long tail cancellation due to optimal threshold Dec 26 15:00:31.075227799 (503h31m30.946310844s) [noise=true , address=77.90.31.11:4580 , piece_id=BB3Q6CFZ4HFFHWWWMPU4K76T4XYNIPTTUZQK6X7QVFUW2XHI4SPQ, seg_pos={0 0}, upload=441864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D): protocol: rpc: dial tcp 77.90.31.11:4580: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:00:31.075237969 (503h31m30.946321014s) [noise=true , address=77.90.31.11:4580 , piece_id=BB3Q6CFZ4HFFHWWWMPU4K76T4XYNIPTTUZQK6X7QVFUW2XHI4SPQ, seg_pos={0 0}, upload=441864]: Long tail cancellation due to optimal threshold Dec 26 15:00:31.075329944 (503h31m30.946412999s) [noise=true , address=145.79.227.30:4787 , piece_id=TCFQCB22TIWKEP4GIKAUS6JOCYHMK6WGRIVYQ7NNS4L5SMPYGUFQ, seg_pos={0 0}, upload=441864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aKqg1aNsv8Yi7fE4asGz8bijcPewWDvUukKcf5SFnzgARju43): protocol: expected piece hash; context canceled; EOF Dec 26 15:00:31.075341406 (503h31m30.946424451s) [noise=true , address=145.79.227.30:4787 , piece_id=TCFQCB22TIWKEP4GIKAUS6JOCYHMK6WGRIVYQ7NNS4L5SMPYGUFQ, seg_pos={0 0}, upload=441864]: Long tail cancellation due to optimal threshold Dec 26 15:00:31.075379379 (503h31m30.946462424s) [seg_pos={0 0}, upload=441864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:00:31.075484318 (503h31m30.946567363s) [upload=441864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:00:31.075491702 (503h31m30.946574737s) [upload=441864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:00:31.376391016 (503h31m31.247474051s) [upload=441864]: Done uploading segments... Dec 26 15:00:31.376412868 (503h31m31.247495903s) [upload=441864]: Upload finished. err: <nil> Duration: 1m0.002335604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X....] duration: 5.785369403s trace: https://jaeger.dev.storj.io/trace/69d78e97cabd159d FAIL (2025-12-26 15:01:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ) context deadline exceeded ======LOGS====== Dec 26 14:58:32.374358736 (503h29m32.245441811s) [upload=441835]: Starting upload Dec 26 14:58:32.374448627 (503h29m32.245531693s) [upload=441835]: Done starting upload Dec 26 14:58:32.374517879 (503h29m32.245600914s) [upload=441835]: Uploading segments... Dec 26 14:58:32.374719773 (503h29m32.245802818s) [upload=441835]: Got next segment. Inline: false Dec 26 14:58:32.374776652 (503h29m32.245859697s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:58:32.374783274 (503h29m32.245866309s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:58:32.623839040 (503h29m32.494922085s) [seg_pos={0 0}, upload=441835]: Begin upload segment... Dec 26 14:58:32.623900226 (503h29m32.494983262s) [seg_pos={0 0}, upload=441835]: Done begin upload segment. Dec 26 14:58:32.623952255 (503h29m32.4950353s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.623959939 (503h29m32.495042975s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624027368 (503h29m32.495110413s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624660373 (503h29m32.495743408s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625232914 (503h29m32.496315959s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625914471 (503h29m32.496997516s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.626458728 (503h29m32.497541773s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627053380 (503h29m32.498136415s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627081644 (503h29m32.498164689s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627849155 (503h29m32.498932191s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628613802 (503h29m32.499696847s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628801959 (503h29m32.499885004s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.629437300 (503h29m32.500520335s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630042502 (503h29m32.501125547s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630237995 (503h29m32.50132104s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630844299 (503h29m32.501927354s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:34.333752309 (503h29m34.204835355s) [upload=441835]: Next returned nil segment Dec 26 14:58:34.333780083 (503h29m34.204863118s) [upload=441835]: Waiting for error group managing segments... Dec 26 14:58:34.629882783 (503h29m34.500965818s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.630250563 (503h29m34.501333598s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.635559923 (503h29m34.506642968s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.637515627 (503h29m34.508598672s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645708077 (503h29m34.516791112s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645931903 (503h29m34.517014948s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.646571351 (503h29m34.517654386s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.652497244 (503h29m34.523580279s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.653752085 (503h29m34.52483513s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.657867030 (503h29m34.528950065s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.664311101 (503h29m34.535394146s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.933887380 (503h29m34.804970415s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950485897 (503h29m34.821568932s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950504161 (503h29m34.821587196s) [seg_pos={0 0}, upload=441835]: Segment reached optimal threshold of 13 pieces. Dec 26 14:58:34.950668044 (503h29m34.821751079s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950673263 (503h29m34.821756298s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950767593 (503h29m34.821850628s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950777642 (503h29m34.821860687s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950845972 (503h29m34.821929007s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950852044 (503h29m34.821935089s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950867774 (503h29m34.821950809s) [seg_pos={0 0}, upload=441835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:58:34.950938940 (503h29m34.822021975s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:58:34.950943258 (503h29m34.822026293s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:58:35.245768465 (503h29m35.116851521s) [upload=441835]: Done uploading segments... Dec 26 14:58:35.245792401 (503h29m35.116875446s) [upload=441835]: Upload finished. err: <nil> Duration: 5.785401024s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 12.18153679s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 15:00:54 UTC) orbiter warning: context deadline exceeded Duration: 12.181565415s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X.] duration: 5.00150443s trace: https://jaeger.dev.storj.io/trace/4a8d7a9fd6b220ce FAIL (2025-12-26 15:00:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001530459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X...] duration: 6.146983899s trace: https://jaeger.dev.storj.io/trace/69d78e97cabd159d FAIL (2025-12-26 15:00:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ) context deadline exceeded ======LOGS====== Dec 26 14:58:32.374358736 (503h29m32.245441811s) [upload=441835]: Starting upload Dec 26 14:58:32.374448627 (503h29m32.245531693s) [upload=441835]: Done starting upload Dec 26 14:58:32.374517879 (503h29m32.245600914s) [upload=441835]: Uploading segments... Dec 26 14:58:32.374719773 (503h29m32.245802818s) [upload=441835]: Got next segment. Inline: false Dec 26 14:58:32.374776652 (503h29m32.245859697s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:58:32.374783274 (503h29m32.245866309s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:58:32.623839040 (503h29m32.494922085s) [seg_pos={0 0}, upload=441835]: Begin upload segment... Dec 26 14:58:32.623900226 (503h29m32.494983262s) [seg_pos={0 0}, upload=441835]: Done begin upload segment. Dec 26 14:58:32.623952255 (503h29m32.4950353s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.623959939 (503h29m32.495042975s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624027368 (503h29m32.495110413s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624660373 (503h29m32.495743408s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625232914 (503h29m32.496315959s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625914471 (503h29m32.496997516s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.626458728 (503h29m32.497541773s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627053380 (503h29m32.498136415s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627081644 (503h29m32.498164689s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627849155 (503h29m32.498932191s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628613802 (503h29m32.499696847s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628801959 (503h29m32.499885004s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.629437300 (503h29m32.500520335s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630042502 (503h29m32.501125547s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630237995 (503h29m32.50132104s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630844299 (503h29m32.501927354s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:34.333752309 (503h29m34.204835355s) [upload=441835]: Next returned nil segment Dec 26 14:58:34.333780083 (503h29m34.204863118s) [upload=441835]: Waiting for error group managing segments... Dec 26 14:58:34.629882783 (503h29m34.500965818s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.630250563 (503h29m34.501333598s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.635559923 (503h29m34.506642968s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.637515627 (503h29m34.508598672s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645708077 (503h29m34.516791112s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645931903 (503h29m34.517014948s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.646571351 (503h29m34.517654386s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.652497244 (503h29m34.523580279s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.653752085 (503h29m34.52483513s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.657867030 (503h29m34.528950065s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.664311101 (503h29m34.535394146s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.933887380 (503h29m34.804970415s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950485897 (503h29m34.821568932s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950504161 (503h29m34.821587196s) [seg_pos={0 0}, upload=441835]: Segment reached optimal threshold of 13 pieces. Dec 26 14:58:34.950668044 (503h29m34.821751079s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950673263 (503h29m34.821756298s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950767593 (503h29m34.821850628s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950777642 (503h29m34.821860687s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950845972 (503h29m34.821929007s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950852044 (503h29m34.821935089s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950867774 (503h29m34.821950809s) [seg_pos={0 0}, upload=441835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:58:34.950938940 (503h29m34.822021975s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:58:34.950943258 (503h29m34.822026293s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:58:35.245768465 (503h29m35.116851521s) [upload=441835]: Done uploading segments... Dec 26 14:58:35.245792401 (503h29m35.116875446s) [upload=441835]: Upload finished. err: <nil> Duration: 6.147021851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XXX] duration: 1m0.003538239s trace: https://jaeger.dev.storj.io/trace/78fa151012f037cc FAIL (2025-12-26 15:00:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.34:4914 , piece_id=336UHAAX7D7477LF3WKA3MM44XSHV75LEABWZZC22RFEPIYUQCRA, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.953711309 (503h30m27.824794354s) [noise=true , address=64.49.13.169:4711 , piece_id=ZOETVL6AKNRI2F747UDRDZZVQHUF66JHSI2RW5THO2GUDMF2HNGQ, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.954009377 (503h30m27.825092412s) [noise=true , address=145.79.227.44:4201 , piece_id=AXFPOZFIRCH3UOOIVDYTD2CYHAFXHM232DIANZWODMGIN2HAVGXA, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.954240286 (503h30m27.825323321s) [noise=true , address=45.91.116.8:4056 , piece_id=EKK4VLXGAPZPCECUNFVPWZ2YG4RCWAVJ4DR6LT2MDAIIJMHUDLPQ, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.954446028 (503h30m27.825529073s) [noise=true , address=77.90.31.21:4893 , piece_id=UB6MT7PAI772N7YY5EOF6M5C2QV3IHUOCMEG6KWBAH7ATQ5EAAMQ, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.954583339 (503h30m27.825666385s) [noise=true , address=77.90.31.11:4564 , piece_id=KL4FUNTBIYXDYLPBAYYWCTMLHFANDPHMR3HRH2W75XXP5FBLIAAA, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.954833606 (503h30m27.825916651s) [noise=true , address=145.79.227.33:4890 , piece_id=LPR5L3N7JCUIC7IYNJMVUJ67WLEZED2T5B5R34ZUAAYWNYIAUIHQ, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.954936622 (503h30m27.826019657s) [noise=true , address=64.49.13.140:4514 , piece_id=N2N3Y4H7BARKN5HXVCTYLQTULQIODSEXQMSEM6HV6BRZVKKOMDTQ, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.955189955 (503h30m27.826273s) [noise=true , address=77.90.31.15:4714 , piece_id=AEQ4MVA4KXQHDP77USBOBRRUAW4JW7BH3L3XZCSUKCTJGYHSG3HA, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.955259498 (503h30m27.826342533s) [noise=true , address=77.90.31.10:4546 , piece_id=J4I22MQT33JPB5JHRQ64JIXN6TKFXLMUCCQ7X6CFTZSS22DTMINQ, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.955605286 (503h30m27.826688321s) [noise=true , address=64.49.13.155:4923 , piece_id=XRB2EL5RDWW7QHNBP3HZS3EFEZJO7YJM4Z7EHQMHCY4PPXCLYAAA, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.955716708 (503h30m27.826799743s) [noise=true , address=45.91.116.9:4271 , piece_id=DILTFWJJHGRCGMLRLCNQZPHYK4ELMSVWX5Q5MM4QRKFL4WHNAUTQ, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.956014605 (503h30m27.82709764s) [noise=true , address=145.79.227.40:4068 , piece_id=OYEDF2ASET5KVQBRI3K5OUPKWUHKAA6KD6OA6UIHEEJH72VXRS5Q, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.956191982 (503h30m27.827275027s) [noise=true , address=145.79.227.30:4768 , piece_id=ZCBXEYHKZXEWI6WHMF5X7MNFSGEBTMNFJOSVEAQRXPKPPY2ZPVTA, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.956435436 (503h30m27.827518491s) [noise=true , address=77.90.31.18:4810 , piece_id=EM2KYFDZ7CWWTDDZKBXLDYRW46B2M7CZLOKV3PJWNBRTGA2BO5PA, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:27.956859494 (503h30m27.827942539s) [noise=true , address=45.91.116.6:4997 , piece_id=QZYHGI45AZUD7D2EGRCMDYJO3PVOZLT4KS2HTVDC7BC74EUMQNUA, seg_pos={0 0}, upload=441849]: Uploading piece... Dec 26 14:59:29.279188896 (503h30m29.150271932s) [upload=441849]: Next returned nil segment Dec 26 14:59:29.279206641 (503h30m29.150289676s) [upload=441849]: Waiting for error group managing segments... Dec 26 14:59:29.614639844 (503h30m29.485722889s) [noise=true , address=64.49.13.155:4923 , piece_id=XRB2EL5RDWW7QHNBP3HZS3EFEZJO7YJM4Z7EHQMHCY4PPXCLYAAA, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:29.826830651 (503h30m29.697913686s) [noise=true , address=145.79.227.43:4162 , piece_id=AHRJVJ4JKG22XPY4ONVP2OU2ZSFC45OI7JB7QXDKY4MQE2RI3GYQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:29.872615264 (503h30m29.743698309s) [noise=true , address=64.49.13.140:4514 , piece_id=N2N3Y4H7BARKN5HXVCTYLQTULQIODSEXQMSEM6HV6BRZVKKOMDTQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.029305479 (503h30m29.900388524s) [noise=true , address=145.79.227.40:4068 , piece_id=OYEDF2ASET5KVQBRI3K5OUPKWUHKAA6KD6OA6UIHEEJH72VXRS5Q, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.062744171 (503h30m29.933827206s) [noise=true , address=77.90.31.18:4810 , piece_id=EM2KYFDZ7CWWTDDZKBXLDYRW46B2M7CZLOKV3PJWNBRTGA2BO5PA, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.096853511 (503h30m29.967936546s) [noise=true , address=45.91.116.8:4056 , piece_id=EKK4VLXGAPZPCECUNFVPWZ2YG4RCWAVJ4DR6LT2MDAIIJMHUDLPQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.126492900 (503h30m29.997575935s) [noise=true , address=45.91.116.9:4271 , piece_id=DILTFWJJHGRCGMLRLCNQZPHYK4ELMSVWX5Q5MM4QRKFL4WHNAUTQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.168802004 (503h30m30.039885039s) [noise=true , address=45.91.116.6:4997 , piece_id=QZYHGI45AZUD7D2EGRCMDYJO3PVOZLT4KS2HTVDC7BC74EUMQNUA, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.212430310 (503h30m30.083513345s) [noise=true , address=77.90.31.15:4714 , piece_id=AEQ4MVA4KXQHDP77USBOBRRUAW4JW7BH3L3XZCSUKCTJGYHSG3HA, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.705974504 (503h30m30.577057539s) [noise=true , address=77.90.31.21:4893 , piece_id=UB6MT7PAI772N7YY5EOF6M5C2QV3IHUOCMEG6KWBAH7ATQ5EAAMQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.708647414 (503h30m30.579730459s) [noise=true , address=77.90.31.19:4859 , piece_id=4H6WAOUYMQ6FLRZC6QQH4W4D5DSLWTMSJ3LOHLN27LYEKFAEIMJQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.712727442 (503h30m30.583810477s) [noise=true , address=145.79.227.30:4768 , piece_id=ZCBXEYHKZXEWI6WHMF5X7MNFSGEBTMNFJOSVEAQRXPKPPY2ZPVTA, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.806810653 (503h30m30.677893688s) [noise=true , address=77.90.31.10:4546 , piece_id=J4I22MQT33JPB5JHRQ64JIXN6TKFXLMUCCQ7X6CFTZSS22DTMINQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.817857749 (503h30m30.688940784s) [noise=true , address=145.79.227.44:4201 , piece_id=AXFPOZFIRCH3UOOIVDYTD2CYHAFXHM232DIANZWODMGIN2HAVGXA, seg_pos={0 0}, upload=441849]: Done uploading piece. err: <nil> Dec 26 14:59:30.817878719 (503h30m30.688961754s) [seg_pos={0 0}, upload=441849]: Segment reached optimal threshold of 14 pieces. Dec 26 14:59:30.818069722 (503h30m30.689152757s) [noise=true , address=145.79.227.33:4890 , piece_id=LPR5L3N7JCUIC7IYNJMVUJ67WLEZED2T5B5R34ZUAAYWNYIAUIHQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 26 14:59:30.818077006 (503h30m30.689160041s) [noise=true , address=145.79.227.33:4890 , piece_id=LPR5L3N7JCUIC7IYNJMVUJ67WLEZED2T5B5R34ZUAAYWNYIAUIHQ, seg_pos={0 0}, upload=441849]: Long tail cancellation due to optimal threshold Dec 26 14:59:30.818084431 (503h30m30.689167466s) [noise=true , address=77.90.31.11:4564 , piece_id=KL4FUNTBIYXDYLPBAYYWCTMLHFANDPHMR3HRH2W75XXP5FBLIAAA, seg_pos={0 0}, upload=441849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: expected piece hash; context canceled; EOF Dec 26 14:59:30.818092356 (503h30m30.689175401s) [noise=true , address=77.90.31.11:4564 , piece_id=KL4FUNTBIYXDYLPBAYYWCTMLHFANDPHMR3HRH2W75XXP5FBLIAAA, seg_pos={0 0}, upload=441849]: Long tail cancellation due to optimal threshold Dec 26 14:59:30.818203838 (503h30m30.689286883s) [noise=true , address=145.79.227.34:4914 , piece_id=336UHAAX7D7477LF3WKA3MM44XSHV75LEABWZZC22RFEPIYUQCRA, seg_pos={0 0}, upload=441849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWwD62SvgY72yFsUXtZMMvCea2xZkRtq72sB7aAkjHoyJWa6N): protocol: expected piece hash; context canceled; EOF Dec 26 14:59:30.818212193 (503h30m30.689295399s) [noise=true , address=145.79.227.34:4914 , piece_id=336UHAAX7D7477LF3WKA3MM44XSHV75LEABWZZC22RFEPIYUQCRA, seg_pos={0 0}, upload=441849]: Long tail cancellation due to optimal threshold Dec 26 14:59:30.818342041 (503h30m30.689425086s) [noise=true , address=64.49.13.169:4711 , piece_id=ZOETVL6AKNRI2F747UDRDZZVQHUF66JHSI2RW5THO2GUDMF2HNGQ, seg_pos={0 0}, upload=441849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:44792->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:44792->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44792->64.49.13.169:4711: use of closed network connection Dec 26 14:59:30.818348233 (503h30m30.689431268s) [noise=true , address=64.49.13.169:4711 , piece_id=ZOETVL6AKNRI2F747UDRDZZVQHUF66JHSI2RW5THO2GUDMF2HNGQ, seg_pos={0 0}, upload=441849]: Long tail cancellation due to optimal threshold Dec 26 14:59:30.818363422 (503h30m30.689446467s) [seg_pos={0 0}, upload=441849]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 14:59:30.818435359 (503h30m30.689518404s) [upload=441849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:59:30.818440218 (503h30m30.689523263s) [upload=441849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:59:31.118084742 (503h30m30.989167777s) [upload=441849]: Done uploading segments... Dec 26 14:59:31.118111744 (503h30m30.989194779s) [upload=441849]: Upload finished. err: <nil> Duration: 1m0.003553979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X..] duration: 5.872085786s trace: https://jaeger.dev.storj.io/trace/69d78e97cabd159d FAIL (2025-12-26 15:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ) context deadline exceeded ======LOGS====== Dec 26 14:58:32.374358736 (503h29m32.245441811s) [upload=441835]: Starting upload Dec 26 14:58:32.374448627 (503h29m32.245531693s) [upload=441835]: Done starting upload Dec 26 14:58:32.374517879 (503h29m32.245600914s) [upload=441835]: Uploading segments... Dec 26 14:58:32.374719773 (503h29m32.245802818s) [upload=441835]: Got next segment. Inline: false Dec 26 14:58:32.374776652 (503h29m32.245859697s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:58:32.374783274 (503h29m32.245866309s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:58:32.623839040 (503h29m32.494922085s) [seg_pos={0 0}, upload=441835]: Begin upload segment... Dec 26 14:58:32.623900226 (503h29m32.494983262s) [seg_pos={0 0}, upload=441835]: Done begin upload segment. Dec 26 14:58:32.623952255 (503h29m32.4950353s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.623959939 (503h29m32.495042975s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624027368 (503h29m32.495110413s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624660373 (503h29m32.495743408s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625232914 (503h29m32.496315959s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625914471 (503h29m32.496997516s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.626458728 (503h29m32.497541773s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627053380 (503h29m32.498136415s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627081644 (503h29m32.498164689s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627849155 (503h29m32.498932191s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628613802 (503h29m32.499696847s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628801959 (503h29m32.499885004s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.629437300 (503h29m32.500520335s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630042502 (503h29m32.501125547s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630237995 (503h29m32.50132104s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630844299 (503h29m32.501927354s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:34.333752309 (503h29m34.204835355s) [upload=441835]: Next returned nil segment Dec 26 14:58:34.333780083 (503h29m34.204863118s) [upload=441835]: Waiting for error group managing segments... Dec 26 14:58:34.629882783 (503h29m34.500965818s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.630250563 (503h29m34.501333598s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.635559923 (503h29m34.506642968s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.637515627 (503h29m34.508598672s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645708077 (503h29m34.516791112s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645931903 (503h29m34.517014948s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.646571351 (503h29m34.517654386s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.652497244 (503h29m34.523580279s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.653752085 (503h29m34.52483513s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.657867030 (503h29m34.528950065s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.664311101 (503h29m34.535394146s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.933887380 (503h29m34.804970415s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950485897 (503h29m34.821568932s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950504161 (503h29m34.821587196s) [seg_pos={0 0}, upload=441835]: Segment reached optimal threshold of 13 pieces. Dec 26 14:58:34.950668044 (503h29m34.821751079s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950673263 (503h29m34.821756298s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950767593 (503h29m34.821850628s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950777642 (503h29m34.821860687s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950845972 (503h29m34.821929007s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950852044 (503h29m34.821935089s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950867774 (503h29m34.821950809s) [seg_pos={0 0}, upload=441835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:58:34.950938940 (503h29m34.822021975s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:58:34.950943258 (503h29m34.822026293s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:58:35.245768465 (503h29m35.116851521s) [upload=441835]: Done uploading segments... Dec 26 14:58:35.245792401 (503h29m35.116875446s) [upload=441835]: Upload finished. err: <nil> Duration: 5.872133426s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 13.43342927s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 15:00:02 UTC) orbiter warning: context deadline exceeded Duration: 13.433460528s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.X] duration: 5.001031648s trace: https://jaeger.dev.storj.io/trace/4a8d7a9fd6b220ce FAIL (2025-12-26 14:59:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001054582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.XX] duration: 1m0.004146395s trace: https://jaeger.dev.storj.io/trace/7cc367cc8ebe8a84 FAIL (2025-12-26 14:59:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30:4767 , piece_id=OXMGJNKX6GKFMF3ZHZWZI5GW5UVRG53ZPIG7D2NZ2U3V257NLO3A, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.705404370 (503h29m27.576487415s) [noise=true , address=193.149.164.143:4701 , piece_id=QBTKCL324MBFHDWZBNZUXFRQG5OXCPKTY3DF35CIXZJYXUTGF6AA, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.705858785 (503h29m27.57694183s) [noise=true , address=77.90.31.10:4548 , piece_id=PQPSEHJFZUOZIHYTRPGSP4PFRVS54D5RPQJRBQW4BMQMZP336CSQ, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.706298402 (503h29m27.577381448s) [noise=true , address=64.49.13.138:4491 , piece_id=ZDPOWL4JGJHRVX43766DGZYOXDDZTJVW5JHEYFZKGPWLA6CKS3QQ, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.706722630 (503h29m27.577805695s) [noise=true , address=145.79.227.40:4084 , piece_id=DTFMCMWTIXIIASAFMVCUUADQFPLX27D2UWLEEX4YCTPJHXR2ZERQ, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.707142720 (503h29m27.578225765s) [noise=true , address=77.90.31.17:4784 , piece_id=M4TPOK6RAXSKVKR2QLQQOSZUAYICWLNEH5EBS67YSAIPSAP5HYWQ, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.707592086 (503h29m27.578675131s) [noise=true , address=77.90.31.13:4625 , piece_id=YO6BTGVY6TFFFIRH4WQII657JG5BLP755SH625O23W4CUJNXTOWQ, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.709699389 (503h29m27.580782434s) [noise=true , address=77.90.31.11:4584 , piece_id=P4BXJVBHBF4Z7VW4ADCJKSRYMRZPBOFI3442KFSW7X523GPSNXJQ, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.712025527 (503h29m27.583108572s) [noise=true , address=64.49.13.170:4720 , piece_id=XW7SBSBD53D7E56QIK6EJOLK5J3EE6URMC3Z743C6R7KOSPDUVPA, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.713189865 (503h29m27.58427291s) [noise=true , address=145.79.227.34:4929 , piece_id=4P2FTNNUGBSZ45Q2KJFUMOTHYRFUH532C534YW5GGK5KFT76FNKQ, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.715090684 (503h29m27.586173729s) [noise=true , address=77.90.31.21:4900 , piece_id=UTK4UNEC6J4Y7DSIYSDJQ4DO4TRGROYYCLFAP3OJNVN2HWQ4X2VA, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.716848542 (503h29m27.587931587s) [noise=true , address=77.90.31.18:4838 , piece_id=YE46AVT3SVSNQGTTIZY54TG43DVWQJVC4XSXTRJH4HBRPNT4EW2A, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.718613583 (503h29m27.589696638s) [noise=true , address=77.90.31.19:4855 , piece_id=HWPCOLLHUTKQ4Q7TJLT3YQFZ3EVF46YQOFQWPR2AHPONT3HAXQRQ, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.720445240 (503h29m27.591528285s) [noise=true , address=145.79.227.44:4222 , piece_id=CHZBEYWILUBLK27PJ25UJXRBO2CAMSHOB5QVHKHL77WPMSEHAU5A, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:27.722257220 (503h29m27.593340265s) [noise=true , address=77.90.31.12:4611 , piece_id=2F6ZQVKYT2L5LBHH44PCQ3FO2LRORVZ3CDXMPXMGI4VSIEJR6RPA, seg_pos={0 0}, upload=441834]: Uploading piece... Dec 26 14:58:29.009498219 (503h29m28.880581264s) [upload=441834]: Next returned nil segment Dec 26 14:58:29.009516024 (503h29m28.880599059s) [upload=441834]: Waiting for error group managing segments... Dec 26 14:58:29.354849548 (503h29m29.225932593s) [noise=true , address=64.49.13.170:4720 , piece_id=XW7SBSBD53D7E56QIK6EJOLK5J3EE6URMC3Z743C6R7KOSPDUVPA, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:29.590505368 (503h29m29.461588413s) [noise=true , address=145.79.227.15:4879 , piece_id=2BZ45NGRIWW76E4K4B77L4RNHWZNIQ3KAWOLCBVLSIOWWCBV7Y6A, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:29.640075718 (503h29m29.511158763s) [noise=true , address=64.49.13.138:4491 , piece_id=ZDPOWL4JGJHRVX43766DGZYOXDDZTJVW5JHEYFZKGPWLA6CKS3QQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:29.752719612 (503h29m29.623802657s) [noise=true , address=193.149.164.143:4701 , piece_id=QBTKCL324MBFHDWZBNZUXFRQG5OXCPKTY3DF35CIXZJYXUTGF6AA, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:29.901086428 (503h29m29.772169473s) [noise=true , address=77.90.31.19:4855 , piece_id=HWPCOLLHUTKQ4Q7TJLT3YQFZ3EVF46YQOFQWPR2AHPONT3HAXQRQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:29.907185992 (503h29m29.778269027s) [noise=true , address=77.90.31.13:4625 , piece_id=YO6BTGVY6TFFFIRH4WQII657JG5BLP755SH625O23W4CUJNXTOWQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:29.962166154 (503h29m29.833249199s) [noise=true , address=77.90.31.12:4611 , piece_id=2F6ZQVKYT2L5LBHH44PCQ3FO2LRORVZ3CDXMPXMGI4VSIEJR6RPA, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:29.970780878 (503h29m29.841863913s) [noise=true , address=77.90.31.21:4900 , piece_id=UTK4UNEC6J4Y7DSIYSDJQ4DO4TRGROYYCLFAP3OJNVN2HWQ4X2VA, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:30.059923040 (503h29m29.931006085s) [noise=true , address=77.90.31.17:4784 , piece_id=M4TPOK6RAXSKVKR2QLQQOSZUAYICWLNEH5EBS67YSAIPSAP5HYWQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:30.505205645 (503h29m30.376288691s) [noise=true , address=45.91.116.4:4959 , piece_id=JAAALZXTHFYVH73D5KY4CB656RYWGPSYUS52J3MEQOO6IE6VO6FQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:30.532690942 (503h29m30.403773987s) [noise=true , address=145.79.227.30:4767 , piece_id=OXMGJNKX6GKFMF3ZHZWZI5GW5UVRG53ZPIG7D2NZ2U3V257NLO3A, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:30.628864002 (503h29m30.499947037s) [noise=true , address=145.79.227.40:4084 , piece_id=DTFMCMWTIXIIASAFMVCUUADQFPLX27D2UWLEEX4YCTPJHXR2ZERQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:30.671640356 (503h29m30.542723391s) [noise=true , address=145.79.227.34:4929 , piece_id=4P2FTNNUGBSZ45Q2KJFUMOTHYRFUH532C534YW5GGK5KFT76FNKQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:30.678194335 (503h29m30.54927737s) [noise=true , address=145.79.227.44:4222 , piece_id=CHZBEYWILUBLK27PJ25UJXRBO2CAMSHOB5QVHKHL77WPMSEHAU5A, seg_pos={0 0}, upload=441834]: Done uploading piece. err: <nil> Dec 26 14:58:30.678216328 (503h29m30.549299373s) [seg_pos={0 0}, upload=441834]: Segment reached optimal threshold of 14 pieces. Dec 26 14:58:30.678389618 (503h29m30.549472653s) [noise=true , address=64.49.13.169:4710 , piece_id=4YY44JEMWOVI64Y5XRXKVRM5SN5FP746Q7TZOQ3UH7JGFE7SSCJA, seg_pos={0 0}, upload=441834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:47816->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:47816->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47816->64.49.13.169:4710: use of closed network connection Dec 26 14:58:30.678396050 (503h29m30.549479085s) [noise=true , address=64.49.13.169:4710 , piece_id=4YY44JEMWOVI64Y5XRXKVRM5SN5FP746Q7TZOQ3UH7JGFE7SSCJA, seg_pos={0 0}, upload=441834]: Long tail cancellation due to optimal threshold Dec 26 14:58:30.678414003 (503h29m30.549497038s) [noise=true , address=77.90.31.18:4838 , piece_id=YE46AVT3SVSNQGTTIZY54TG43DVWQJVC4XSXTRJH4HBRPNT4EW2A, seg_pos={0 0}, upload=441834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:30.678422220 (503h29m30.549505265s) [noise=true , address=77.90.31.18:4838 , piece_id=YE46AVT3SVSNQGTTIZY54TG43DVWQJVC4XSXTRJH4HBRPNT4EW2A, seg_pos={0 0}, upload=441834]: Long tail cancellation due to optimal threshold Dec 26 14:58:30.678596391 (503h29m30.549679436s) [noise=true , address=77.90.31.11:4584 , piece_id=P4BXJVBHBF4Z7VW4ADCJKSRYMRZPBOFI3442KFSW7X523GPSNXJQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:42704->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:42704->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42704->77.90.31.11:4584: use of closed network connection Dec 26 14:58:30.678606410 (503h29m30.549689455s) [noise=true , address=77.90.31.11:4584 , piece_id=P4BXJVBHBF4Z7VW4ADCJKSRYMRZPBOFI3442KFSW7X523GPSNXJQ, seg_pos={0 0}, upload=441834]: Long tail cancellation due to optimal threshold Dec 26 14:58:30.678743361 (503h29m30.549826396s) [noise=true , address=77.90.31.10:4548 , piece_id=PQPSEHJFZUOZIHYTRPGSP4PFRVS54D5RPQJRBQW4BMQMZP336CSQ, seg_pos={0 0}, upload=441834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:30.678751496 (503h29m30.549834541s) [noise=true , address=77.90.31.10:4548 , piece_id=PQPSEHJFZUOZIHYTRPGSP4PFRVS54D5RPQJRBQW4BMQMZP336CSQ, seg_pos={0 0}, upload=441834]: Long tail cancellation due to optimal threshold Dec 26 14:58:30.678769421 (503h29m30.549852466s) [seg_pos={0 0}, upload=441834]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 14:58:30.678851006 (503h29m30.549934051s) [upload=441834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:58:30.678859642 (503h29m30.549942677s) [upload=441834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:58:30.966544835 (503h29m30.83762787s) [upload=441834]: Done uploading segments... Dec 26 14:58:30.966564933 (503h29m30.837647968s) [upload=441834]: Upload finished. err: <nil> Duration: 1m0.004159232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X.] duration: 5.823399089s trace: https://jaeger.dev.storj.io/trace/69d78e97cabd159d FAIL (2025-12-26 14:59:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ) context deadline exceeded ======LOGS====== Dec 26 14:58:32.374358736 (503h29m32.245441811s) [upload=441835]: Starting upload Dec 26 14:58:32.374448627 (503h29m32.245531693s) [upload=441835]: Done starting upload Dec 26 14:58:32.374517879 (503h29m32.245600914s) [upload=441835]: Uploading segments... Dec 26 14:58:32.374719773 (503h29m32.245802818s) [upload=441835]: Got next segment. Inline: false Dec 26 14:58:32.374776652 (503h29m32.245859697s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:58:32.374783274 (503h29m32.245866309s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:58:32.623839040 (503h29m32.494922085s) [seg_pos={0 0}, upload=441835]: Begin upload segment... Dec 26 14:58:32.623900226 (503h29m32.494983262s) [seg_pos={0 0}, upload=441835]: Done begin upload segment. Dec 26 14:58:32.623952255 (503h29m32.4950353s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.623959939 (503h29m32.495042975s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624027368 (503h29m32.495110413s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624660373 (503h29m32.495743408s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625232914 (503h29m32.496315959s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625914471 (503h29m32.496997516s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.626458728 (503h29m32.497541773s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627053380 (503h29m32.498136415s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627081644 (503h29m32.498164689s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627849155 (503h29m32.498932191s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628613802 (503h29m32.499696847s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628801959 (503h29m32.499885004s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.629437300 (503h29m32.500520335s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630042502 (503h29m32.501125547s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630237995 (503h29m32.50132104s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630844299 (503h29m32.501927354s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:34.333752309 (503h29m34.204835355s) [upload=441835]: Next returned nil segment Dec 26 14:58:34.333780083 (503h29m34.204863118s) [upload=441835]: Waiting for error group managing segments... Dec 26 14:58:34.629882783 (503h29m34.500965818s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.630250563 (503h29m34.501333598s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.635559923 (503h29m34.506642968s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.637515627 (503h29m34.508598672s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645708077 (503h29m34.516791112s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645931903 (503h29m34.517014948s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.646571351 (503h29m34.517654386s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.652497244 (503h29m34.523580279s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.653752085 (503h29m34.52483513s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.657867030 (503h29m34.528950065s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.664311101 (503h29m34.535394146s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.933887380 (503h29m34.804970415s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950485897 (503h29m34.821568932s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950504161 (503h29m34.821587196s) [seg_pos={0 0}, upload=441835]: Segment reached optimal threshold of 13 pieces. Dec 26 14:58:34.950668044 (503h29m34.821751079s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950673263 (503h29m34.821756298s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950767593 (503h29m34.821850628s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950777642 (503h29m34.821860687s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950845972 (503h29m34.821929007s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950852044 (503h29m34.821935089s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950867774 (503h29m34.821950809s) [seg_pos={0 0}, upload=441835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:58:34.950938940 (503h29m34.822021975s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:58:34.950943258 (503h29m34.822026293s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:58:35.245768465 (503h29m35.116851521s) [upload=441835]: Done uploading segments... Dec 26 14:58:35.245792401 (503h29m35.116875446s) [upload=441835]: Upload finished. err: <nil> Duration: 5.823417854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.......X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X.X] duration: 30.004382778s trace: https://jaeger.dev.storj.io/trace/69d78e97cabd159d FAIL (2025-12-26 14:59:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ) context deadline exceeded ======LOGS====== Dec 26 14:58:32.374358736 (503h29m32.245441811s) [upload=441835]: Starting upload Dec 26 14:58:32.374448627 (503h29m32.245531693s) [upload=441835]: Done starting upload Dec 26 14:58:32.374517879 (503h29m32.245600914s) [upload=441835]: Uploading segments... Dec 26 14:58:32.374719773 (503h29m32.245802818s) [upload=441835]: Got next segment. Inline: false Dec 26 14:58:32.374776652 (503h29m32.245859697s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:58:32.374783274 (503h29m32.245866309s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:58:32.623839040 (503h29m32.494922085s) [seg_pos={0 0}, upload=441835]: Begin upload segment... Dec 26 14:58:32.623900226 (503h29m32.494983262s) [seg_pos={0 0}, upload=441835]: Done begin upload segment. Dec 26 14:58:32.623952255 (503h29m32.4950353s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.623959939 (503h29m32.495042975s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624027368 (503h29m32.495110413s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.624660373 (503h29m32.495743408s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625232914 (503h29m32.496315959s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.625914471 (503h29m32.496997516s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.626458728 (503h29m32.497541773s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627053380 (503h29m32.498136415s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627081644 (503h29m32.498164689s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.627849155 (503h29m32.498932191s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628613802 (503h29m32.499696847s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.628801959 (503h29m32.499885004s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.629437300 (503h29m32.500520335s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630042502 (503h29m32.501125547s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630237995 (503h29m32.50132104s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:32.630844299 (503h29m32.501927354s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Uploading piece... Dec 26 14:58:34.333752309 (503h29m34.204835355s) [upload=441835]: Next returned nil segment Dec 26 14:58:34.333780083 (503h29m34.204863118s) [upload=441835]: Waiting for error group managing segments... Dec 26 14:58:34.629882783 (503h29m34.500965818s) [noise=true , address=145.79.227.22:4511 , piece_id=BXX2VGENOPMG35BYGUZ4A3XCD7P6W7IDQUST7JXO5UPWTRSHKOVA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.630250563 (503h29m34.501333598s) [noise=true , address=145.79.227.49:4338 , piece_id=QX6OWE576NHLFTUOVW626FG2GW2CEVWMTXKNUW6FY7QW2I7G5D7A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.635559923 (503h29m34.506642968s) [noise=true , address=145.79.227.11:4729 , piece_id=JTQWUT3GRL6CGFYIFGXWOXCKZWJGSRSROECP47TRR2JUI6STKRQQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.637515627 (503h29m34.508598672s) [noise=true , address=145.79.227.37:4567 , piece_id=UAUJFECLHDSNONZG7EZ6GZHUU3QSOWUQHZMSHQNFLWP2UP6RJPXQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645708077 (503h29m34.516791112s) [noise=true , address=145.79.227.39:4013 , piece_id=KMKVZRDV46AAA6ZY74TDPUCETIJRCERAXQLYSBPZE2FEDL65WMIQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.645931903 (503h29m34.517014948s) [noise=true , address=145.79.227.47:4265 , piece_id=RJNP7EYIYRYKZ4QLIBHC4JBI3UQLNDM7AJC7SMS5IKFRPZ5PLDPQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.646571351 (503h29m34.517654386s) [noise=true , address=145.79.227.3:4150 , piece_id=RZBSJ7BZ47LWNTYLWRPTQGV3PPXU7S6QKLWVFNW7QIH3GTHSEW4A, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.652497244 (503h29m34.523580279s) [noise=true , address=145.79.227.35:4482 , piece_id=F3IWJIXHDWZYQI2CI2QIUZD5YJFUQ4ZR7DTWFQVEOTXYICJMQ5VQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.653752085 (503h29m34.52483513s) [noise=true , address=145.79.227.13:4804 , piece_id=V2WXPFYD5GCQQYV2DRBKT44KJLCQ6SAZCQB5Z3A7326FZTDR7XEA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.657867030 (503h29m34.528950065s) [noise=true , address=145.79.227.17:4971 , piece_id=XDYMXN22D36OTQMLWDNVWQMSJBWR45FZXX2DYXAFXRGFOQJQLLYA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.664311101 (503h29m34.535394146s) [noise=true , address=145.79.227.38:4978 , piece_id=S34F3J4PBUZQHBPRCM77YQJSPXFEHVLQW43YE4AVIZ2NKJAKH4HA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.933887380 (503h29m34.804970415s) [noise=true , address=77.90.31.17:4773 , piece_id=4ASLBTL3QTNVNRAVKJTBCHL3CM66RO7ZVUL5E7GSY4RAUOVIJWVQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950485897 (503h29m34.821568932s) [noise=true , address=45.91.116.14:4190 , piece_id=TWYVQDT5ZSQRUQGSVJCDDBODISPKE7O425VX6AOPPXHHWUBFC6PQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: <nil> Dec 26 14:58:34.950504161 (503h29m34.821587196s) [seg_pos={0 0}, upload=441835]: Segment reached optimal threshold of 13 pieces. Dec 26 14:58:34.950668044 (503h29m34.821751079s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950673263 (503h29m34.821756298s) [noise=true , address=145.79.227.40:4049 , piece_id=RYDM5NV4TEVJNYBUSL26U6FUYOUBCHM5RU6ZIQQ7AXVFJ7F7L7MA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950767593 (503h29m34.821850628s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950777642 (503h29m34.821860687s) [noise=true , address=77.90.31.14:4662 , piece_id=SE2IAAWT6SFYE2URH7RNF6DGWSNMN3NUOD72TQ25AGSS3ERXDSGQ, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950845972 (503h29m34.821929007s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 14:58:34.950852044 (503h29m34.821935089s) [noise=true , address=77.90.31.10:4517 , piece_id=U6OSW5ROJALQD27S2AFKLURHLEZIPCCZVU2W7PI32GYFQEP2CCPA, seg_pos={0 0}, upload=441835]: Long tail cancellation due to optimal threshold Dec 26 14:58:34.950867774 (503h29m34.821950809s) [seg_pos={0 0}, upload=441835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:58:34.950938940 (503h29m34.822021975s) [upload=441835]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:58:34.950943258 (503h29m34.822026293s) [upload=441835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:58:35.245768465 (503h29m35.116851521s) [upload=441835]: Done uploading segments... Dec 26 14:58:35.245792401 (503h29m35.116875446s) [upload=441835]: Upload finished. err: <nil> Duration: 30.004409258s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 16.119252009s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:58:52 UTC) orbiter warning: context deadline exceeded Duration: 16.119280674s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX.] duration: 3.782883664s trace: https://jaeger.dev.storj.io/trace/33756f2f44731ad6 FAIL (2025-12-26 14:58:52 UTC) orbiter warning: context deadline exceeded Duration: 3.782907449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.X] duration: 1m0.00298227s trace: https://jaeger.dev.storj.io/trace/46d2a4cb410b109c FAIL (2025-12-26 14:58:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, Piece ID: DMZQY5VL3WBWKD3DL5JPI3TZ2EJ4V2THN2UXR7HUZHEL65N3PPSA) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 2XQ7GUMT3NLULL2MLKQ5MUXM3HF66EVL2DPVG6B6YMMUAZFKWJCA) context deadline exceeded; piecestore: (Node ID: 1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, Piece ID: P4CSC4AGB72LH2VH6H3RDQALCOWXNHM235IHLYHTV7SJB7BIRGIA) context deadline exceeded; piecestore: (Node ID: 1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, Piece ID: 6SU7MUKBMJK3KPMXQYOAZHZXP6MBZZ6PYABIHN6LZEYYCRP34MGA) context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4218 , piece_id=I35Q7IX6FM5V3KOFXELR2F7LYEJAVNFDFQJOS77OS3YQY2CLTMFA, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.455090046 (503h28m27.326173081s) [noise=true , address=77.90.31.15:4711 , piece_id=ITQTU6NQY7G65X6Q3MAJWC3EL2TLQB3H36FA7HO4FKVT3Y2YXYTA, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.455676362 (503h28m27.326759397s) [noise=true , address=77.90.31.13:4633 , piece_id=P7P26VJNGPUTLULVEEQWXO7QXB3KPNL6HGSJW5USVTPLBQFGVUMQ, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.455897503 (503h28m27.326980538s) [noise=true , address=145.79.227.40:4078 , piece_id=IEBHR2P4IMJB4AZMAUTSQPHSK53R3SR6KISJ36SCCESE4QZ5MNSA, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.456176154 (503h28m27.327259199s) [noise=true , address=145.79.227.30:4777 , piece_id=2IFFXS2XXAB6WZTRFLQZSUXH5AAEG6NNTOPMPKR6CGYL3EVLZ5CQ, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.456559123 (503h28m27.327642168s) [noise=true , address=145.79.227.33:4894 , piece_id=5OIPGGNIORCOLZIFRSIDO3PAPNW6EF7HYOWVBNVGOUB2KZEPZFXQ, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.456610732 (503h28m27.327693767s) [noise=true , address=145.79.227.34:4912 , piece_id=XLIPHXJ5WHXLSDXZ3566JMILYZV26XCSPLDKGG4FYA5MH6KEPPOA, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.456964124 (503h28m27.328047159s) [noise=true , address=77.90.31.10:4542 , piece_id=6SU7MUKBMJK3KPMXQYOAZHZXP6MBZZ6PYABIHN6LZEYYCRP34MGA, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.457067612 (503h28m27.328150647s) [noise=true , address=77.90.31.12:4599 , piece_id=J2LYIB6HGWOBPO2UYPFQL3UB4FA3V6VYV3QJ2DCZV6YZ6ORP4W4Q, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.457502329 (503h28m27.328585364s) [noise=true , address=77.90.31.11:4559 , piece_id=P4CSC4AGB72LH2VH6H3RDQALCOWXNHM235IHLYHTV7SJB7BIRGIA, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.457715335 (503h28m27.32879838s) [noise=true , address=77.90.31.16:4764 , piece_id=P27DZOJOX5TSHU6GMVMKATKF6RYAVZZCP637KO7KNKJBIFCR5TTQ, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.457917870 (503h28m27.329000905s) [noise=true , address=77.90.31.21:4891 , piece_id=YNBBPSCKH7JJ33AA5HKFECN7AT6ALOEVU3GRH7JVFMP6A7EOKPEQ, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.458221408 (503h28m27.329304443s) [noise=true , address=77.90.31.18:4822 , piece_id=DMZQY5VL3WBWKD3DL5JPI3TZ2EJ4V2THN2UXR7HUZHEL65N3PPSA, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.458348511 (503h28m27.329431556s) [noise=true , address=45.91.116.6:4988 , piece_id=NFSJNABC24TUIUCIX3EDHRBJZWXCN2OTEZ7RJHYG4AQKQFTH6JZQ, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.458896224 (503h28m27.329979269s) [noise=true , address=64.49.13.169:4705 , piece_id=XGZXM6PKCKZRQCTIHA36YTWQYHCXCQAAFK4RM4A7CPTKNPJREYPQ, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:27.459382350 (503h28m27.330465395s) [noise=true , address=64.49.13.139:4474 , piece_id=QEHOOWRB5TPUH4HZDMEF3DMWROGHCU73AFGKGCFFAFCHKX3KMJAA, seg_pos={0 0}, upload=441817]: Uploading piece... Dec 26 14:57:28.781581745 (503h28m28.65266479s) [upload=441817]: Next returned nil segment Dec 26 14:57:28.781595771 (503h28m28.652678816s) [upload=441817]: Waiting for error group managing segments... Dec 26 14:57:29.119220888 (503h28m28.990303933s) [noise=true , address=64.49.13.139:4474 , piece_id=QEHOOWRB5TPUH4HZDMEF3DMWROGHCU73AFGKGCFFAFCHKX3KMJAA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:29.318294799 (503h28m29.189377844s) [noise=true , address=145.79.227.14:4855 , piece_id=W5DW5WVTF3Y4ZB4EIK6P6RGNNVIH2D7AWTL7UGKSVFCUOI3ZJRPQ, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:29.593503274 (503h28m29.464586319s) [noise=true , address=77.90.31.21:4891 , piece_id=YNBBPSCKH7JJ33AA5HKFECN7AT6ALOEVU3GRH7JVFMP6A7EOKPEQ, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:29.673026519 (503h28m29.544109565s) [noise=true , address=45.91.116.7:4043 , piece_id=2XQ7GUMT3NLULL2MLKQ5MUXM3HF66EVL2DPVG6B6YMMUAZFKWJCA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:29.707509562 (503h28m29.578592607s) [noise=true , address=45.91.116.6:4988 , piece_id=NFSJNABC24TUIUCIX3EDHRBJZWXCN2OTEZ7RJHYG4AQKQFTH6JZQ, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:29.712980198 (503h28m29.584063234s) [noise=true , address=77.90.31.15:4711 , piece_id=ITQTU6NQY7G65X6Q3MAJWC3EL2TLQB3H36FA7HO4FKVT3Y2YXYTA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:29.735053049 (503h28m29.606136094s) [noise=true , address=77.90.31.12:4599 , piece_id=J2LYIB6HGWOBPO2UYPFQL3UB4FA3V6VYV3QJ2DCZV6YZ6ORP4W4Q, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:30.206114070 (503h28m30.077197105s) [noise=true , address=77.90.31.18:4822 , piece_id=DMZQY5VL3WBWKD3DL5JPI3TZ2EJ4V2THN2UXR7HUZHEL65N3PPSA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:30.260333875 (503h28m30.13141692s) [noise=true , address=145.79.227.30:4777 , piece_id=2IFFXS2XXAB6WZTRFLQZSUXH5AAEG6NNTOPMPKR6CGYL3EVLZ5CQ, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:30.269901001 (503h28m30.140984036s) [noise=true , address=145.79.227.44:4218 , piece_id=I35Q7IX6FM5V3KOFXELR2F7LYEJAVNFDFQJOS77OS3YQY2CLTMFA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:30.276140954 (503h28m30.147223989s) [noise=true , address=145.79.227.40:4078 , piece_id=IEBHR2P4IMJB4AZMAUTSQPHSK53R3SR6KISJ36SCCESE4QZ5MNSA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:30.420596513 (503h28m30.291679558s) [noise=true , address=77.90.31.10:4542 , piece_id=6SU7MUKBMJK3KPMXQYOAZHZXP6MBZZ6PYABIHN6LZEYYCRP34MGA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:30.473862290 (503h28m30.344945335s) [noise=true , address=145.79.227.33:4894 , piece_id=5OIPGGNIORCOLZIFRSIDO3PAPNW6EF7HYOWVBNVGOUB2KZEPZFXQ, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:30.529676690 (503h28m30.400759725s) [noise=true , address=77.90.31.11:4559 , piece_id=P4CSC4AGB72LH2VH6H3RDQALCOWXNHM235IHLYHTV7SJB7BIRGIA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: <nil> Dec 26 14:57:30.529697199 (503h28m30.400780234s) [seg_pos={0 0}, upload=441817]: Segment reached optimal threshold of 14 pieces. Dec 26 14:57:30.529918059 (503h28m30.401001104s) [noise=true , address=77.90.31.16:4764 , piece_id=P27DZOJOX5TSHU6GMVMKATKF6RYAVZZCP637KO7KNKJBIFCR5TTQ, seg_pos={0 0}, upload=441817]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT): protocol: expected piece hash; context canceled; EOF Dec 26 14:57:30.529925203 (503h28m30.401008248s) [noise=true , address=77.90.31.16:4764 , piece_id=P27DZOJOX5TSHU6GMVMKATKF6RYAVZZCP637KO7KNKJBIFCR5TTQ, seg_pos={0 0}, upload=441817]: Long tail cancellation due to optimal threshold Dec 26 14:57:30.530226467 (503h28m30.401309502s) [noise=true , address=145.79.227.34:4912 , piece_id=XLIPHXJ5WHXLSDXZ3566JMILYZV26XCSPLDKGG4FYA5MH6KEPPOA, seg_pos={0 0}, upload=441817]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x): protocol: expected piece hash; context canceled; EOF Dec 26 14:57:30.530235103 (503h28m30.401318158s) [noise=true , address=145.79.227.34:4912 , piece_id=XLIPHXJ5WHXLSDXZ3566JMILYZV26XCSPLDKGG4FYA5MH6KEPPOA, seg_pos={0 0}, upload=441817]: Long tail cancellation due to optimal threshold Dec 26 14:57:30.530359370 (503h28m30.401442415s) [noise=true , address=64.49.13.169:4705 , piece_id=XGZXM6PKCKZRQCTIHA36YTWQYHCXCQAAFK4RM4A7CPTKNPJREYPQ, seg_pos={0 0}, upload=441817]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57518->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57518->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57518->64.49.13.169:4705: use of closed network connection Dec 26 14:57:30.530365481 (503h28m30.401448516s) [noise=true , address=64.49.13.169:4705 , piece_id=XGZXM6PKCKZRQCTIHA36YTWQYHCXCQAAFK4RM4A7CPTKNPJREYPQ, seg_pos={0 0}, upload=441817]: Long tail cancellation due to optimal threshold Dec 26 14:57:30.530482995 (503h28m30.40156604s) [noise=true , address=77.90.31.13:4633 , piece_id=P7P26VJNGPUTLULVEEQWXO7QXB3KPNL6HGSJW5USVTPLBQFGVUMQ, seg_pos={0 0}, upload=441817]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF Dec 26 14:57:30.530490168 (503h28m30.401573203s) [noise=true , address=77.90.31.13:4633 , piece_id=P7P26VJNGPUTLULVEEQWXO7QXB3KPNL6HGSJW5USVTPLBQFGVUMQ, seg_pos={0 0}, upload=441817]: Long tail cancellation due to optimal threshold Dec 26 14:57:30.530538431 (503h28m30.401621466s) [seg_pos={0 0}, upload=441817]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 14:57:30.530616910 (503h28m30.401699945s) [upload=441817]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:57:30.530621569 (503h28m30.401704614s) [upload=441817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:57:30.831992300 (503h28m30.703075355s) [upload=441817]: Done uploading segments... Dec 26 14:57:30.832021155 (503h28m30.7031042s) [upload=441817]: Upload finished. err: <nil> Duration: 1m0.002996137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [........X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X.] duration: 5.763297314s trace: https://jaeger.dev.storj.io/trace/73e2004131e7b0b8 FAIL (2025-12-26 14:58:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: U7YQ2FVOHDB5ZMPK52Z34GQX7R3CBYRHRTJIYQCP2QJJSE4NJEXQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 5RBFQ4KP6ILN6HN2MANAVXSX2C2C4L2JZE3LUL2KZDHI76P2QBRQ) context deadline exceeded ======LOGS====== Dec 26 14:57:43.234760102 (503h28m43.105843137s) [upload=441822]: Starting upload Dec 26 14:57:43.234817051 (503h28m43.105900086s) [upload=441822]: Done starting upload Dec 26 14:57:43.234956165 (503h28m43.10603921s) [upload=441822]: Uploading segments... Dec 26 14:57:43.235044423 (503h28m43.106127488s) [upload=441822]: Got next segment. Inline: false Dec 26 14:57:43.235101662 (503h28m43.106184707s) [upload=441822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:57:43.235108114 (503h28m43.106191159s) [upload=441822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:57:43.478446552 (503h28m43.349529598s) [seg_pos={0 0}, upload=441822]: Begin upload segment... Dec 26 14:57:43.478509503 (503h28m43.349592538s) [seg_pos={0 0}, upload=441822]: Done begin upload segment. Dec 26 14:57:43.478520474 (503h28m43.349603519s) [noise=true , address=145.79.227.48:4302 , piece_id=GFYVAZ5HAUK3SW3RUCLLT4DKBKCH5VFJ2SCCAILPACZLS2WDBX4A, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.478550330 (503h28m43.349633375s) [noise=true , address=45.91.116.6:4974 , piece_id=TXORMKPTQWPXCWBRILPIOJ3ZTUBSIFPFCX2PIQBQ4LPUAKU3HRRA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.478610445 (503h28m43.34969349s) [noise=true , address=77.90.31.16:4736 , piece_id=5RBFQ4KP6ILN6HN2MANAVXSX2C2C4L2JZE3LUL2KZDHI76P2QBRQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.478991551 (503h28m43.350074596s) [noise=true , address=145.79.227.41:4086 , piece_id=4GJOGLS2EYNPQXTLREEFQNW7BMEFI6IGRJD3TPUU3M6DXF4CZQHQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.479844315 (503h28m43.35092736s) [noise=true , address=145.79.227.6:4257 , piece_id=6J44F3UJ3L6NBWOWP5SKUIJQFDL3A435N7WG4PRS64NFU22TXB4A, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.480221823 (503h28m43.351304868s) [noise=true , address=145.79.227.47:4266 , piece_id=2JI4JLRFJZPQ2C446LCCZNIXI5BZRBYGCKA7P7OCE56BGJFJD4JA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.480559266 (503h28m43.351642301s) [noise=true , address=145.79.227.2:4114 , piece_id=EEZAQNYMIPK6W4Z5EGLYHZMN7BVKMFHX3HCWQV3DT3XELSTIKBBQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.480909372 (503h28m43.351992407s) [noise=true , address=45.91.116.5:4297 , piece_id=XGQ6TEIY72GPENPOFRXLMHUFQORZBTMSPCOHB4MDS64P7BMLHYCQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.481377805 (503h28m43.35246084s) [noise=true , address=145.79.227.42:4121 , piece_id=HKK2EVER6F3PZTUF5Q2IEQTMWGZFV2CZ5HQ24LKK7YCZCXYGEE7A, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.481580521 (503h28m43.352663566s) [noise=true , address=145.79.227.16:4912 , piece_id=KIPAG6OLUOOLRVD3FHPYK6I5K7UQVO7LVPWJKS2GWCAN4RP67KCA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.482319849 (503h28m43.353402894s) [noise=true , address=145.79.227.27:4691 , piece_id=NMDDCSVI3XGVDQ4E42M5ENPIJIIWRCLKW3MKSTEF4LKVHFRTBDZA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.483565270 (503h28m43.354648305s) [noise=true , address=45.91.116.11:4120 , piece_id=U7YQ2FVOHDB5ZMPK52Z34GQX7R3CBYRHRTJIYQCP2QJJSE4NJEXQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.484596834 (503h28m43.355679869s) [noise=true , address=145.79.227.11:4729 , piece_id=BEBKYK3YJMWVIUFBRKRUBKPYWHQJBX4JNHFVQNG3LSPYYRHW23SQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.485784485 (503h28m43.35686753s) [noise=true , address=145.79.227.50:4375 , piece_id=PD6KEMJE6PTJATBTPJXE5TDHW2JDLKXENB4WXCCEFAETDZ57SZZA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.486984651 (503h28m43.358067696s) [noise=true , address=145.79.227.46:4602 , piece_id=SY5736UBHS2DFX6UENPRMUEUCFSTVZ6T3PO63QJ5O5ZXRUW673VA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.488342036 (503h28m43.359425071s) [noise=true , address=45.91.116.14:4191 , piece_id=IGCFHXPNP73PIKZIWEFOGMMBAPQ7LRMCLIQ5D2X7JNBDVWXWM47Q, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:45.219597244 (503h28m45.090680279s) [upload=441822]: Next returned nil segment Dec 26 14:57:45.219613736 (503h28m45.090696781s) [upload=441822]: Waiting for error group managing segments... Dec 26 14:57:45.479348115 (503h28m45.35043116s) [noise=true , address=145.79.227.48:4302 , piece_id=GFYVAZ5HAUK3SW3RUCLLT4DKBKCH5VFJ2SCCAILPACZLS2WDBX4A, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.488724670 (503h28m45.359807705s) [noise=true , address=145.79.227.42:4121 , piece_id=HKK2EVER6F3PZTUF5Q2IEQTMWGZFV2CZ5HQ24LKK7YCZCXYGEE7A, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.490090491 (503h28m45.361173536s) [noise=true , address=145.79.227.6:4257 , piece_id=6J44F3UJ3L6NBWOWP5SKUIJQFDL3A435N7WG4PRS64NFU22TXB4A, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.490418415 (503h28m45.36150145s) [noise=true , address=145.79.227.2:4114 , piece_id=EEZAQNYMIPK6W4Z5EGLYHZMN7BVKMFHX3HCWQV3DT3XELSTIKBBQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.493092677 (503h28m45.364175722s) [noise=true , address=145.79.227.46:4602 , piece_id=SY5736UBHS2DFX6UENPRMUEUCFSTVZ6T3PO63QJ5O5ZXRUW673VA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.493713740 (503h28m45.364796775s) [noise=true , address=145.79.227.27:4691 , piece_id=NMDDCSVI3XGVDQ4E42M5ENPIJIIWRCLKW3MKSTEF4LKVHFRTBDZA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.494526227 (503h28m45.365609262s) [noise=true , address=145.79.227.50:4375 , piece_id=PD6KEMJE6PTJATBTPJXE5TDHW2JDLKXENB4WXCCEFAETDZ57SZZA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.499747830 (503h28m45.370830865s) [noise=true , address=145.79.227.47:4266 , piece_id=2JI4JLRFJZPQ2C446LCCZNIXI5BZRBYGCKA7P7OCE56BGJFJD4JA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.504809118 (503h28m45.375892153s) [noise=true , address=145.79.227.41:4086 , piece_id=4GJOGLS2EYNPQXTLREEFQNW7BMEFI6IGRJD3TPUU3M6DXF4CZQHQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.509231579 (503h28m45.380314634s) [noise=true , address=145.79.227.16:4912 , piece_id=KIPAG6OLUOOLRVD3FHPYK6I5K7UQVO7LVPWJKS2GWCAN4RP67KCA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.509938775 (503h28m45.38102181s) [noise=true , address=145.79.227.11:4729 , piece_id=BEBKYK3YJMWVIUFBRKRUBKPYWHQJBX4JNHFVQNG3LSPYYRHW23SQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.807285164 (503h28m45.678368199s) [noise=true , address=77.90.31.16:4736 , piece_id=5RBFQ4KP6ILN6HN2MANAVXSX2C2C4L2JZE3LUL2KZDHI76P2QBRQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.827919876 (503h28m45.699002912s) [noise=true , address=45.91.116.11:4120 , piece_id=U7YQ2FVOHDB5ZMPK52Z34GQX7R3CBYRHRTJIYQCP2QJJSE4NJEXQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.827941126 (503h28m45.699024161s) [seg_pos={0 0}, upload=441822]: Segment reached optimal threshold of 13 pieces. Dec 26 14:57:45.828223654 (503h28m45.699306699s) [noise=true , address=45.91.116.5:4297 , piece_id=XGQ6TEIY72GPENPOFRXLMHUFQORZBTMSPCOHB4MDS64P7BMLHYCQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 14:57:45.828233794 (503h28m45.699316829s) [noise=true , address=45.91.116.5:4297 , piece_id=XGQ6TEIY72GPENPOFRXLMHUFQORZBTMSPCOHB4MDS64P7BMLHYCQ, seg_pos={0 0}, upload=441822]: Long tail cancellation due to optimal threshold Dec 26 14:57:45.828816043 (503h28m45.699899088s) [noise=true , address=45.91.116.6:4974 , piece_id=TXORMKPTQWPXCWBRILPIOJ3ZTUBSIFPFCX2PIQBQ4LPUAKU3HRRA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: read tcp 188.166.225.170:42180->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 14:57:45.828826032 (503h28m45.699909067s) [noise=true , address=45.91.116.6:4974 , piece_id=TXORMKPTQWPXCWBRILPIOJ3ZTUBSIFPFCX2PIQBQ4LPUAKU3HRRA, seg_pos={0 0}, upload=441822]: Long tail cancellation due to optimal threshold Dec 26 14:57:45.829046972 (503h28m45.700130017s) [noise=true , address=45.91.116.14:4191 , piece_id=IGCFHXPNP73PIKZIWEFOGMMBAPQ7LRMCLIQ5D2X7JNBDVWXWM47Q, seg_pos={0 0}, upload=441822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 14:57:45.829057061 (503h28m45.700140096s) [noise=true , address=45.91.116.14:4191 , piece_id=IGCFHXPNP73PIKZIWEFOGMMBAPQ7LRMCLIQ5D2X7JNBDVWXWM47Q, seg_pos={0 0}, upload=441822]: Long tail cancellation due to optimal threshold Dec 26 14:57:45.829080296 (503h28m45.700163331s) [seg_pos={0 0}, upload=441822]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:57:45.829242355 (503h28m45.70032539s) [upload=441822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:57:45.829250570 (503h28m45.700333605s) [upload=441822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:57:46.128083580 (503h28m45.999166615s) [upload=441822]: Done uploading segments... Dec 26 14:57:46.128106443 (503h28m45.999189488s) [upload=441822]: Upload finished. err: <nil> Duration: 5.763313553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.........X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..X] duration: 30.00358648s trace: https://jaeger.dev.storj.io/trace/73e2004131e7b0b8 FAIL (2025-12-26 14:58:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: U7YQ2FVOHDB5ZMPK52Z34GQX7R3CBYRHRTJIYQCP2QJJSE4NJEXQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 5RBFQ4KP6ILN6HN2MANAVXSX2C2C4L2JZE3LUL2KZDHI76P2QBRQ) context deadline exceeded ======LOGS====== Dec 26 14:57:43.234760102 (503h28m43.105843137s) [upload=441822]: Starting upload Dec 26 14:57:43.234817051 (503h28m43.105900086s) [upload=441822]: Done starting upload Dec 26 14:57:43.234956165 (503h28m43.10603921s) [upload=441822]: Uploading segments... Dec 26 14:57:43.235044423 (503h28m43.106127488s) [upload=441822]: Got next segment. Inline: false Dec 26 14:57:43.235101662 (503h28m43.106184707s) [upload=441822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:57:43.235108114 (503h28m43.106191159s) [upload=441822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:57:43.478446552 (503h28m43.349529598s) [seg_pos={0 0}, upload=441822]: Begin upload segment... Dec 26 14:57:43.478509503 (503h28m43.349592538s) [seg_pos={0 0}, upload=441822]: Done begin upload segment. Dec 26 14:57:43.478520474 (503h28m43.349603519s) [noise=true , address=145.79.227.48:4302 , piece_id=GFYVAZ5HAUK3SW3RUCLLT4DKBKCH5VFJ2SCCAILPACZLS2WDBX4A, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.478550330 (503h28m43.349633375s) [noise=true , address=45.91.116.6:4974 , piece_id=TXORMKPTQWPXCWBRILPIOJ3ZTUBSIFPFCX2PIQBQ4LPUAKU3HRRA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.478610445 (503h28m43.34969349s) [noise=true , address=77.90.31.16:4736 , piece_id=5RBFQ4KP6ILN6HN2MANAVXSX2C2C4L2JZE3LUL2KZDHI76P2QBRQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.478991551 (503h28m43.350074596s) [noise=true , address=145.79.227.41:4086 , piece_id=4GJOGLS2EYNPQXTLREEFQNW7BMEFI6IGRJD3TPUU3M6DXF4CZQHQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.479844315 (503h28m43.35092736s) [noise=true , address=145.79.227.6:4257 , piece_id=6J44F3UJ3L6NBWOWP5SKUIJQFDL3A435N7WG4PRS64NFU22TXB4A, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.480221823 (503h28m43.351304868s) [noise=true , address=145.79.227.47:4266 , piece_id=2JI4JLRFJZPQ2C446LCCZNIXI5BZRBYGCKA7P7OCE56BGJFJD4JA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.480559266 (503h28m43.351642301s) [noise=true , address=145.79.227.2:4114 , piece_id=EEZAQNYMIPK6W4Z5EGLYHZMN7BVKMFHX3HCWQV3DT3XELSTIKBBQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.480909372 (503h28m43.351992407s) [noise=true , address=45.91.116.5:4297 , piece_id=XGQ6TEIY72GPENPOFRXLMHUFQORZBTMSPCOHB4MDS64P7BMLHYCQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.481377805 (503h28m43.35246084s) [noise=true , address=145.79.227.42:4121 , piece_id=HKK2EVER6F3PZTUF5Q2IEQTMWGZFV2CZ5HQ24LKK7YCZCXYGEE7A, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.481580521 (503h28m43.352663566s) [noise=true , address=145.79.227.16:4912 , piece_id=KIPAG6OLUOOLRVD3FHPYK6I5K7UQVO7LVPWJKS2GWCAN4RP67KCA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.482319849 (503h28m43.353402894s) [noise=true , address=145.79.227.27:4691 , piece_id=NMDDCSVI3XGVDQ4E42M5ENPIJIIWRCLKW3MKSTEF4LKVHFRTBDZA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.483565270 (503h28m43.354648305s) [noise=true , address=45.91.116.11:4120 , piece_id=U7YQ2FVOHDB5ZMPK52Z34GQX7R3CBYRHRTJIYQCP2QJJSE4NJEXQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.484596834 (503h28m43.355679869s) [noise=true , address=145.79.227.11:4729 , piece_id=BEBKYK3YJMWVIUFBRKRUBKPYWHQJBX4JNHFVQNG3LSPYYRHW23SQ, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.485784485 (503h28m43.35686753s) [noise=true , address=145.79.227.50:4375 , piece_id=PD6KEMJE6PTJATBTPJXE5TDHW2JDLKXENB4WXCCEFAETDZ57SZZA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.486984651 (503h28m43.358067696s) [noise=true , address=145.79.227.46:4602 , piece_id=SY5736UBHS2DFX6UENPRMUEUCFSTVZ6T3PO63QJ5O5ZXRUW673VA, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:43.488342036 (503h28m43.359425071s) [noise=true , address=45.91.116.14:4191 , piece_id=IGCFHXPNP73PIKZIWEFOGMMBAPQ7LRMCLIQ5D2X7JNBDVWXWM47Q, seg_pos={0 0}, upload=441822]: Uploading piece... Dec 26 14:57:45.219597244 (503h28m45.090680279s) [upload=441822]: Next returned nil segment Dec 26 14:57:45.219613736 (503h28m45.090696781s) [upload=441822]: Waiting for error group managing segments... Dec 26 14:57:45.479348115 (503h28m45.35043116s) [noise=true , address=145.79.227.48:4302 , piece_id=GFYVAZ5HAUK3SW3RUCLLT4DKBKCH5VFJ2SCCAILPACZLS2WDBX4A, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.488724670 (503h28m45.359807705s) [noise=true , address=145.79.227.42:4121 , piece_id=HKK2EVER6F3PZTUF5Q2IEQTMWGZFV2CZ5HQ24LKK7YCZCXYGEE7A, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.490090491 (503h28m45.361173536s) [noise=true , address=145.79.227.6:4257 , piece_id=6J44F3UJ3L6NBWOWP5SKUIJQFDL3A435N7WG4PRS64NFU22TXB4A, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.490418415 (503h28m45.36150145s) [noise=true , address=145.79.227.2:4114 , piece_id=EEZAQNYMIPK6W4Z5EGLYHZMN7BVKMFHX3HCWQV3DT3XELSTIKBBQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.493092677 (503h28m45.364175722s) [noise=true , address=145.79.227.46:4602 , piece_id=SY5736UBHS2DFX6UENPRMUEUCFSTVZ6T3PO63QJ5O5ZXRUW673VA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.493713740 (503h28m45.364796775s) [noise=true , address=145.79.227.27:4691 , piece_id=NMDDCSVI3XGVDQ4E42M5ENPIJIIWRCLKW3MKSTEF4LKVHFRTBDZA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.494526227 (503h28m45.365609262s) [noise=true , address=145.79.227.50:4375 , piece_id=PD6KEMJE6PTJATBTPJXE5TDHW2JDLKXENB4WXCCEFAETDZ57SZZA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.499747830 (503h28m45.370830865s) [noise=true , address=145.79.227.47:4266 , piece_id=2JI4JLRFJZPQ2C446LCCZNIXI5BZRBYGCKA7P7OCE56BGJFJD4JA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.504809118 (503h28m45.375892153s) [noise=true , address=145.79.227.41:4086 , piece_id=4GJOGLS2EYNPQXTLREEFQNW7BMEFI6IGRJD3TPUU3M6DXF4CZQHQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.509231579 (503h28m45.380314634s) [noise=true , address=145.79.227.16:4912 , piece_id=KIPAG6OLUOOLRVD3FHPYK6I5K7UQVO7LVPWJKS2GWCAN4RP67KCA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.509938775 (503h28m45.38102181s) [noise=true , address=145.79.227.11:4729 , piece_id=BEBKYK3YJMWVIUFBRKRUBKPYWHQJBX4JNHFVQNG3LSPYYRHW23SQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.807285164 (503h28m45.678368199s) [noise=true , address=77.90.31.16:4736 , piece_id=5RBFQ4KP6ILN6HN2MANAVXSX2C2C4L2JZE3LUL2KZDHI76P2QBRQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.827919876 (503h28m45.699002912s) [noise=true , address=45.91.116.11:4120 , piece_id=U7YQ2FVOHDB5ZMPK52Z34GQX7R3CBYRHRTJIYQCP2QJJSE4NJEXQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: <nil> Dec 26 14:57:45.827941126 (503h28m45.699024161s) [seg_pos={0 0}, upload=441822]: Segment reached optimal threshold of 13 pieces. Dec 26 14:57:45.828223654 (503h28m45.699306699s) [noise=true , address=45.91.116.5:4297 , piece_id=XGQ6TEIY72GPENPOFRXLMHUFQORZBTMSPCOHB4MDS64P7BMLHYCQ, seg_pos={0 0}, upload=441822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 14:57:45.828233794 (503h28m45.699316829s) [noise=true , address=45.91.116.5:4297 , piece_id=XGQ6TEIY72GPENPOFRXLMHUFQORZBTMSPCOHB4MDS64P7BMLHYCQ, seg_pos={0 0}, upload=441822]: Long tail cancellation due to optimal threshold Dec 26 14:57:45.828816043 (503h28m45.699899088s) [noise=true , address=45.91.116.6:4974 , piece_id=TXORMKPTQWPXCWBRILPIOJ3ZTUBSIFPFCX2PIQBQ4LPUAKU3HRRA, seg_pos={0 0}, upload=441822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: read tcp 188.166.225.170:42180->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 14:57:45.828826032 (503h28m45.699909067s) [noise=true , address=45.91.116.6:4974 , piece_id=TXORMKPTQWPXCWBRILPIOJ3ZTUBSIFPFCX2PIQBQ4LPUAKU3HRRA, seg_pos={0 0}, upload=441822]: Long tail cancellation due to optimal threshold Dec 26 14:57:45.829046972 (503h28m45.700130017s) [noise=true , address=45.91.116.14:4191 , piece_id=IGCFHXPNP73PIKZIWEFOGMMBAPQ7LRMCLIQ5D2X7JNBDVWXWM47Q, seg_pos={0 0}, upload=441822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 14:57:45.829057061 (503h28m45.700140096s) [noise=true , address=45.91.116.14:4191 , piece_id=IGCFHXPNP73PIKZIWEFOGMMBAPQ7LRMCLIQ5D2X7JNBDVWXWM47Q, seg_pos={0 0}, upload=441822]: Long tail cancellation due to optimal threshold Dec 26 14:57:45.829080296 (503h28m45.700163331s) [seg_pos={0 0}, upload=441822]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:57:45.829242355 (503h28m45.70032539s) [upload=441822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:57:45.829250570 (503h28m45.700333605s) [upload=441822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:57:46.128083580 (503h28m45.999166615s) [upload=441822]: Done uploading segments... Dec 26 14:57:46.128106443 (503h28m45.999189488s) [upload=441822]: Upload finished. err: <nil> Duration: 30.003616699s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 8.19839014s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:58:12 UTC) orbiter warning: context deadline exceeded Duration: 8.198419556s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXXX] duration: 5.001147008s trace: https://jaeger.dev.storj.io/trace/33756f2f44731ad6 FAIL (2025-12-26 14:58:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001171124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 15.101197688s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:57:56 UTC) orbiter warning: context deadline exceeded Duration: 15.101244057s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXXX] duration: 5.001767139s trace: https://jaeger.dev.storj.io/trace/6bed8c2abc4a6ede FAIL (2025-12-26 14:57:42 UTC) orbiter warning: context deadline exceeded Duration: 5.00178924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X.X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX.] duration: 57.494902324s trace: https://jaeger.dev.storj.io/trace/25943d7d6e21b101 FAIL (2025-12-26 14:57:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4BKBGUTXPZBUEOD4MUYM4MMUUSZIIVYPF5JNYRA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.191896815 (503h26m21.06297985s) [noise=true , address=45.91.116.7:4013 , piece_id=VKXKKL6FPFYJXLM567GM2FKHYW7PKVV35YGWENO4GLVD2KR63TDA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.192253004 (503h26m21.063336039s) [noise=true , address=45.91.116.8:4056 , piece_id=ISDYXCRZPW26GEJDM2VZXR6A4XU4PXB2WUQGF3DHXMT44DGADYYQ, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.192538788 (503h26m21.063621833s) [noise=true , address=145.79.227.30:4787 , piece_id=F24XWZ57566E7G4X4OXNGI5XM343CH7IIR3QHUKBDJXIEOIZDYBA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.192825544 (503h26m21.063908589s) [noise=true , address=145.79.227.44:4227 , piece_id=CS2BGGIHE5BU6DKYWMEUODVX65Y6XKMN4LTQBNQOVVJN4BJTAEWA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.193283065 (503h26m21.06436611s) [noise=true , address=193.149.164.143:4694 , piece_id=6FI7UBV7WEPBQADMWMZHDHBDPG75WJSF7Y3ZWCNJ5MDHYL6RGIMA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.193694899 (503h26m21.064777934s) [noise=true , address=77.90.31.17:4790 , piece_id=3Y4YIEZSSXGZJQATWMFBNJIVKASIV2QDE5SAYFYA4WHJHZHJGJWA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.194087366 (503h26m21.065170411s) [noise=true , address=77.90.31.21:4910 , piece_id=BLPPM6QRNLPFIGIGBZ7X2IAZNW5JNSVQ6WOF5LAB32IBOYQZABNA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.194512546 (503h26m21.065595591s) [noise=true , address=145.79.227.43:4162 , piece_id=DIT24Q2HJLNDOHCWBEGDTCMFY5QJXXHUKRPPJF2BGJQ2H3SLPWNA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.194814160 (503h26m21.065897205s) [noise=true , address=77.90.31.16:4751 , piece_id=5SV6BRZC2QGJ4NGPJQU6IRUVW56FTFYONW7R5BEICQVK3O3EPGLQ, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.195184426 (503h26m21.066267471s) [noise=true , address=45.91.116.12:4238 , piece_id=27VXSSQUTDFRGIIN4K2IEAVWIM7DQWQJ2YPKYJGFDT7IHRTWS5ZA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.195473225 (503h26m21.066556261s) [noise=true , address=77.90.31.14:4693 , piece_id=OLPVPOZRUXHJ564V4T7ZJHAGVVOPI7IAV5TQZF72BUJGTQWRMAPA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.195872536 (503h26m21.066955571s) [noise=true , address=145.79.227.40:4057 , piece_id=IK2FBGDQMOSARSMQQTD76YRTMQ6K6AALNOEVIKI43HULEOGN5TUA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.196181714 (503h26m21.067264759s) [noise=true , address=77.90.31.18:4835 , piece_id=6GV2HPDN3IHJN6VEIKNEOOLP4K424FAMKAY4ANZSUHQWVSTYY3YA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.196556859 (503h26m21.067639894s) [noise=true , address=77.90.31.13:4633 , piece_id=PK62PUCWFHSI4DKUAB5PN4J4LSPG6DMJQRRG6QZPA37ZRGUZYWJA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.196893440 (503h26m21.067976475s) [noise=true , address=77.90.31.15:4715 , piece_id=GMHFOJ75XG6LXUZTU5PBBH6F54FYPI4GTZME7FSCUD4SOUWS7TZQ, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.197348967 (503h26m21.068432002s) [noise=true , address=45.91.116.4:4965 , piece_id=X5OSQPIG7VZVZE463ITVIYLOTM2HROOLLUDOYMPKKRKK6Q7ZXVIA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:22.510105822 (503h26m22.381188857s) [upload=441787]: Next returned nil segment Dec 26 14:55:22.510122724 (503h26m22.381205769s) [upload=441787]: Waiting for error group managing segments... Dec 26 14:55:22.844880193 (503h26m22.715963238s) [noise=true , address=193.149.164.143:4694 , piece_id=6FI7UBV7WEPBQADMWMZHDHBDPG75WJSF7Y3ZWCNJ5MDHYL6RGIMA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.232317072 (503h26m23.103400107s) [noise=true , address=77.90.31.18:4835 , piece_id=6GV2HPDN3IHJN6VEIKNEOOLP4K424FAMKAY4ANZSUHQWVSTYY3YA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.278192617 (503h26m23.149275662s) [noise=true , address=77.90.31.17:4790 , piece_id=3Y4YIEZSSXGZJQATWMFBNJIVKASIV2QDE5SAYFYA4WHJHZHJGJWA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.328005339 (503h26m23.199088384s) [noise=true , address=45.91.116.4:4965 , piece_id=X5OSQPIG7VZVZE463ITVIYLOTM2HROOLLUDOYMPKKRKK6Q7ZXVIA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.341675700 (503h26m23.212758745s) [noise=true , address=77.90.31.11:4571 , piece_id=QFFMLCHHLYMQ34BKBGUTXPZBUEOD4MUYM4MMUUSZIIVYPF5JNYRA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.362265206 (503h26m23.233348252s) [noise=true , address=77.90.31.12:4609 , piece_id=BXZJIBBUTJUMS5QXMUO5GJK4FZBLQ5DDBSPQCDLWJSMWBX63K4SQ, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.385788289 (503h26m23.256871324s) [noise=true , address=77.90.31.13:4633 , piece_id=PK62PUCWFHSI4DKUAB5PN4J4LSPG6DMJQRRG6QZPA37ZRGUZYWJA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.471028862 (503h26m23.342111897s) [noise=true , address=77.90.31.21:4910 , piece_id=BLPPM6QRNLPFIGIGBZ7X2IAZNW5JNSVQ6WOF5LAB32IBOYQZABNA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.571884980 (503h26m23.442968015s) [noise=true , address=77.90.31.16:4751 , piece_id=5SV6BRZC2QGJ4NGPJQU6IRUVW56FTFYONW7R5BEICQVK3O3EPGLQ, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.830143060 (503h26m23.701226105s) [noise=true , address=45.91.116.8:4056 , piece_id=ISDYXCRZPW26GEJDM2VZXR6A4XU4PXB2WUQGF3DHXMT44DGADYYQ, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.937329342 (503h26m23.808412387s) [noise=true , address=77.90.31.15:4715 , piece_id=GMHFOJ75XG6LXUZTU5PBBH6F54FYPI4GTZME7FSCUD4SOUWS7TZQ, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.977987703 (503h26m23.849070748s) [noise=true , address=145.79.227.44:4227 , piece_id=CS2BGGIHE5BU6DKYWMEUODVX65Y6XKMN4LTQBNQOVVJN4BJTAEWA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.996396166 (503h26m23.867479211s) [noise=true , address=145.79.227.30:4787 , piece_id=F24XWZ57566E7G4X4OXNGI5XM343CH7IIR3QHUKBDJXIEOIZDYBA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:24.024734728 (503h26m23.895817773s) [noise=true , address=145.79.227.40:4057 , piece_id=IK2FBGDQMOSARSMQQTD76YRTMQ6K6AALNOEVIKI43HULEOGN5TUA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:24.024758894 (503h26m23.895841939s) [seg_pos={0 0}, upload=441787]: Segment reached optimal threshold of 14 pieces. Dec 26 14:55:24.024969955 (503h26m23.89605299s) [noise=true , address=77.90.31.14:4693 , piece_id=OLPVPOZRUXHJ564V4T7ZJHAGVVOPI7IAV5TQZF72BUJGTQWRMAPA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:24.024978663 (503h26m23.896061698s) [noise=true , address=77.90.31.14:4693 , piece_id=OLPVPOZRUXHJ564V4T7ZJHAGVVOPI7IAV5TQZF72BUJGTQWRMAPA, seg_pos={0 0}, upload=441787]: Long tail cancellation due to optimal threshold Dec 26 14:55:24.025516927 (503h26m23.896599962s) [noise=true , address=45.91.116.12:4238 , piece_id=27VXSSQUTDFRGIIN4K2IEAVWIM7DQWQJ2YPKYJGFDT7IHRTWS5ZA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:24.025524672 (503h26m23.896607707s) [noise=true , address=45.91.116.12:4238 , piece_id=27VXSSQUTDFRGIIN4K2IEAVWIM7DQWQJ2YPKYJGFDT7IHRTWS5ZA, seg_pos={0 0}, upload=441787]: Long tail cancellation due to optimal threshold Dec 26 14:55:24.025635954 (503h26m23.896718989s) [noise=true , address=145.79.227.43:4162 , piece_id=DIT24Q2HJLNDOHCWBEGDTCMFY5QJXXHUKRPPJF2BGJQ2H3SLPWNA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:24.025641064 (503h26m23.896724099s) [noise=true , address=145.79.227.43:4162 , piece_id=DIT24Q2HJLNDOHCWBEGDTCMFY5QJXXHUKRPPJF2BGJQ2H3SLPWNA, seg_pos={0 0}, upload=441787]: Long tail cancellation due to optimal threshold Dec 26 14:55:24.025896971 (503h26m23.896980006s) [noise=true , address=45.91.116.7:4013 , piece_id=VKXKKL6FPFYJXLM567GM2FKHYW7PKVV35YGWENO4GLVD2KR63TDA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: read tcp 188.166.225.170:57120->45.91.116.7:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 14:55:24.025904505 (503h26m23.89698755s) [noise=true , address=45.91.116.7:4013 , piece_id=VKXKKL6FPFYJXLM567GM2FKHYW7PKVV35YGWENO4GLVD2KR63TDA, seg_pos={0 0}, upload=441787]: Long tail cancellation due to optimal threshold Dec 26 14:55:24.025919745 (503h26m23.89700279s) [seg_pos={0 0}, upload=441787]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 14:55:24.025990900 (503h26m23.897073935s) [upload=441787]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:55:24.025996771 (503h26m23.897079806s) [upload=441787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:55:24.333961281 (503h26m24.205044316s) [upload=441787]: Done uploading segments... Dec 26 14:55:24.333985157 (503h26m24.205068202s) [upload=441787]: Upload finished. err: <nil> Duration: 57.494926259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 7.527890714s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:57:20 UTC) orbiter warning: context deadline exceeded Duration: 7.527916985s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..........X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX..] duration: 5.863072722s trace: https://jaeger.dev.storj.io/trace/28fef441e93b4d1b FAIL (2025-12-26 14:57:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA) context deadline exceeded ======LOGS====== 26 14:55:15.481986730 (503h26m15.353069775s) [upload=441785]: Starting upload Dec 26 14:55:15.482086199 (503h26m15.353169255s) [upload=441785]: Done starting upload Dec 26 14:55:15.482211829 (503h26m15.353294874s) [upload=441785]: Uploading segments... Dec 26 14:55:15.482374409 (503h26m15.353457444s) [upload=441785]: Got next segment. Inline: false Dec 26 14:55:15.482417651 (503h26m15.353500686s) [upload=441785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:55:15.482425215 (503h26m15.35350826s) [upload=441785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:55:15.732804392 (503h26m15.603887447s) [seg_pos={0 0}, upload=441785]: Begin upload segment... Dec 26 14:55:15.732849418 (503h26m15.603932463s) [seg_pos={0 0}, upload=441785]: Done begin upload segment. Dec 26 14:55:15.732884695 (503h26m15.603967741s) [noise=true , address=145.79.227.12:4767 , piece_id=MDTY5FYUYZA5ELZGAAIHDDSNR3BJGAUANQ2FVIZWFWFAT7AU7R2Q, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.732943978 (503h26m15.604027023s) [noise=true , address=145.79.227.37:4566 , piece_id=QLB2F7PN6H3DOEJB644KUS3HNFXZBF232SDACNOEM6BS56JSFZSQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.733486110 (503h26m15.604569155s) [noise=true , address=145.79.227.42:4123 , piece_id=APGOKWFBAUFVCEFVIQ4KOXXYRT5RP4OFDROBG5HSC6ZOXQDPMIDQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.733839894 (503h26m15.604922939s) [noise=true , address=145.79.227.47:4267 , piece_id=RG6ENDWL4VELIUA2JI6BLGQ3ZDN2NMRU7J7ICBS37SLZFXKZXLOA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734163019 (503h26m15.605246054s) [noise=true , address=145.79.227.49:4337 , piece_id=LGUPZGZLOQ2DB6GRHVLA6KLLJOMF5XZKC5ICYPBHF6C6UZ7WO5VQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734457751 (503h26m15.605540796s) [noise=true , address=145.79.227.17:4971 , piece_id=4N5FT4Y77VM7L5J43YIZ6KIYDFZK2VVCP64XLA62X4UONR7Y6HVA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734762611 (503h26m15.605845646s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.735058594 (503h26m15.60614163s) [noise=true , address=45.91.116.11:4120 , piece_id=NSCOJSHLZSVJEQIJJ7IH4KWCDNRM5YUHYQ4TGBOGYXNHJT76VRVQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.735470689 (503h26m15.606553734s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.737056249 (503h26m15.608139294s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.737213398 (503h26m15.608296433s) [noise=true , address=145.79.227.21:4473 , piece_id=U7NGK72WQ5T4DBOZ4GC2Z4BL2OOHY2ZUU7JGCS77CGVTUU6ZNLRA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.738567907 (503h26m15.609650942s) [noise=true , address=145.79.227.19:4078 , piece_id=7LWXRMLUQLKLH3RMLAILN4N6VIKBFH57257RSQVOA2MMC4EFIPAQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.739954838 (503h26m15.611037883s) [noise=true , address=45.91.116.10:4084 , piece_id=2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.741117191 (503h26m15.612200236s) [noise=true , address=45.91.116.12:4227 , piece_id=4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.742502920 (503h26m15.613585955s) [noise=true , address=145.79.227.6:4257 , piece_id=THHD7N43R5OPTGS4A6FW47F2YXXV7TM3RJBIH7VA7WOUFXFQRWKQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.743815189 (503h26m15.614898224s) [noise=true , address=45.91.116.5:4300 , piece_id=PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:17.443210343 (503h26m17.314293388s) [upload=441785]: Next returned nil segment Dec 26 14:55:17.443224791 (503h26m17.314307836s) [upload=441785]: Waiting for error group managing segments... Dec 26 14:55:17.742099611 (503h26m17.613182656s) [noise=true , address=145.79.227.19:4078 , piece_id=7LWXRMLUQLKLH3RMLAILN4N6VIKBFH57257RSQVOA2MMC4EFIPAQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.743202762 (503h26m17.614285807s) [noise=true , address=145.79.227.42:4123 , piece_id=APGOKWFBAUFVCEFVIQ4KOXXYRT5RP4OFDROBG5HSC6ZOXQDPMIDQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.749438656 (503h26m17.620521701s) [noise=true , address=145.79.227.21:4473 , piece_id=U7NGK72WQ5T4DBOZ4GC2Z4BL2OOHY2ZUU7JGCS77CGVTUU6ZNLRA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.753956048 (503h26m17.625039093s) [noise=true , address=145.79.227.47:4267 , piece_id=RG6ENDWL4VELIUA2JI6BLGQ3ZDN2NMRU7J7ICBS37SLZFXKZXLOA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.755334604 (503h26m17.626417639s) [noise=true , address=145.79.227.37:4566 , piece_id=QLB2F7PN6H3DOEJB644KUS3HNFXZBF232SDACNOEM6BS56JSFZSQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.755766065 (503h26m17.6268491s) [noise=true , address=145.79.227.49:4337 , piece_id=LGUPZGZLOQ2DB6GRHVLA6KLLJOMF5XZKC5ICYPBHF6C6UZ7WO5VQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.758197936 (503h26m17.629280971s) [noise=true , address=145.79.227.6:4257 , piece_id=THHD7N43R5OPTGS4A6FW47F2YXXV7TM3RJBIH7VA7WOUFXFQRWKQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.766691289 (503h26m17.637774334s) [noise=true , address=145.79.227.12:4767 , piece_id=MDTY5FYUYZA5ELZGAAIHDDSNR3BJGAUANQ2FVIZWFWFAT7AU7R2Q, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.852019669 (503h26m17.723102714s) [noise=true , address=145.79.227.17:4971 , piece_id=4N5FT4Y77VM7L5J43YIZ6KIYDFZK2VVCP64XLA62X4UONR7Y6HVA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.985792848 (503h26m17.856875883s) [noise=true , address=45.91.116.10:4084 , piece_id=2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.105553042 (503h26m17.976636088s) [noise=true , address=45.91.116.11:4120 , piece_id=NSCOJSHLZSVJEQIJJ7IH4KWCDNRM5YUHYQ4TGBOGYXNHJT76VRVQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.173726066 (503h26m18.044809101s) [noise=true , address=45.91.116.12:4227 , piece_id=4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.189895806 (503h26m18.060978851s) [noise=true , address=45.91.116.5:4300 , piece_id=PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.189920062 (503h26m18.061003097s) [seg_pos={0 0}, upload=441785]: Segment reached optimal threshold of 13 pieces. Dec 26 14:55:18.190151133 (503h26m18.061234168s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection Dec 26 14:55:18.190158887 (503h26m18.061241932s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190364599 (503h26m18.061447644s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:18.190403503 (503h26m18.061486548s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190473005 (503h26m18.06155605s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:18.190481852 (503h26m18.061564887s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190503013 (503h26m18.061586058s) [seg_pos={0 0}, upload=441785]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:55:18.190582794 (503h26m18.061665839s) [upload=441785]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:55:18.190588114 (503h26m18.061671159s) [upload=441785]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:55:18.500579403 (503h26m18.371662448s) [upload=441785]: Done uploading segments... Dec 26 14:55:18.500604862 (503h26m18.371687907s) [upload=441785]: Upload finished. err: <nil> Duration: 5.863105134s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXXX] duration: 5.001769653s trace: https://jaeger.dev.storj.io/trace/59645b5f5f9559c2 FAIL (2025-12-26 14:56:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001790453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 13.444448271s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:56:35 UTC) orbiter warning: context deadline exceeded Duration: 13.444471465s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.X.X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...XX] duration: 1m0.002963436s trace: https://jaeger.dev.storj.io/trace/25943d7d6e21b101 FAIL (2025-12-26 14:56:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4BKBGUTXPZBUEOD4MUYM4MMUUSZIIVYPF5JNYRA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.191896815 (503h26m21.06297985s) [noise=true , address=45.91.116.7:4013 , piece_id=VKXKKL6FPFYJXLM567GM2FKHYW7PKVV35YGWENO4GLVD2KR63TDA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.192253004 (503h26m21.063336039s) [noise=true , address=45.91.116.8:4056 , piece_id=ISDYXCRZPW26GEJDM2VZXR6A4XU4PXB2WUQGF3DHXMT44DGADYYQ, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.192538788 (503h26m21.063621833s) [noise=true , address=145.79.227.30:4787 , piece_id=F24XWZ57566E7G4X4OXNGI5XM343CH7IIR3QHUKBDJXIEOIZDYBA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.192825544 (503h26m21.063908589s) [noise=true , address=145.79.227.44:4227 , piece_id=CS2BGGIHE5BU6DKYWMEUODVX65Y6XKMN4LTQBNQOVVJN4BJTAEWA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.193283065 (503h26m21.06436611s) [noise=true , address=193.149.164.143:4694 , piece_id=6FI7UBV7WEPBQADMWMZHDHBDPG75WJSF7Y3ZWCNJ5MDHYL6RGIMA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.193694899 (503h26m21.064777934s) [noise=true , address=77.90.31.17:4790 , piece_id=3Y4YIEZSSXGZJQATWMFBNJIVKASIV2QDE5SAYFYA4WHJHZHJGJWA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.194087366 (503h26m21.065170411s) [noise=true , address=77.90.31.21:4910 , piece_id=BLPPM6QRNLPFIGIGBZ7X2IAZNW5JNSVQ6WOF5LAB32IBOYQZABNA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.194512546 (503h26m21.065595591s) [noise=true , address=145.79.227.43:4162 , piece_id=DIT24Q2HJLNDOHCWBEGDTCMFY5QJXXHUKRPPJF2BGJQ2H3SLPWNA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.194814160 (503h26m21.065897205s) [noise=true , address=77.90.31.16:4751 , piece_id=5SV6BRZC2QGJ4NGPJQU6IRUVW56FTFYONW7R5BEICQVK3O3EPGLQ, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.195184426 (503h26m21.066267471s) [noise=true , address=45.91.116.12:4238 , piece_id=27VXSSQUTDFRGIIN4K2IEAVWIM7DQWQJ2YPKYJGFDT7IHRTWS5ZA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.195473225 (503h26m21.066556261s) [noise=true , address=77.90.31.14:4693 , piece_id=OLPVPOZRUXHJ564V4T7ZJHAGVVOPI7IAV5TQZF72BUJGTQWRMAPA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.195872536 (503h26m21.066955571s) [noise=true , address=145.79.227.40:4057 , piece_id=IK2FBGDQMOSARSMQQTD76YRTMQ6K6AALNOEVIKI43HULEOGN5TUA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.196181714 (503h26m21.067264759s) [noise=true , address=77.90.31.18:4835 , piece_id=6GV2HPDN3IHJN6VEIKNEOOLP4K424FAMKAY4ANZSUHQWVSTYY3YA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.196556859 (503h26m21.067639894s) [noise=true , address=77.90.31.13:4633 , piece_id=PK62PUCWFHSI4DKUAB5PN4J4LSPG6DMJQRRG6QZPA37ZRGUZYWJA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.196893440 (503h26m21.067976475s) [noise=true , address=77.90.31.15:4715 , piece_id=GMHFOJ75XG6LXUZTU5PBBH6F54FYPI4GTZME7FSCUD4SOUWS7TZQ, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:21.197348967 (503h26m21.068432002s) [noise=true , address=45.91.116.4:4965 , piece_id=X5OSQPIG7VZVZE463ITVIYLOTM2HROOLLUDOYMPKKRKK6Q7ZXVIA, seg_pos={0 0}, upload=441787]: Uploading piece... Dec 26 14:55:22.510105822 (503h26m22.381188857s) [upload=441787]: Next returned nil segment Dec 26 14:55:22.510122724 (503h26m22.381205769s) [upload=441787]: Waiting for error group managing segments... Dec 26 14:55:22.844880193 (503h26m22.715963238s) [noise=true , address=193.149.164.143:4694 , piece_id=6FI7UBV7WEPBQADMWMZHDHBDPG75WJSF7Y3ZWCNJ5MDHYL6RGIMA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.232317072 (503h26m23.103400107s) [noise=true , address=77.90.31.18:4835 , piece_id=6GV2HPDN3IHJN6VEIKNEOOLP4K424FAMKAY4ANZSUHQWVSTYY3YA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.278192617 (503h26m23.149275662s) [noise=true , address=77.90.31.17:4790 , piece_id=3Y4YIEZSSXGZJQATWMFBNJIVKASIV2QDE5SAYFYA4WHJHZHJGJWA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.328005339 (503h26m23.199088384s) [noise=true , address=45.91.116.4:4965 , piece_id=X5OSQPIG7VZVZE463ITVIYLOTM2HROOLLUDOYMPKKRKK6Q7ZXVIA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.341675700 (503h26m23.212758745s) [noise=true , address=77.90.31.11:4571 , piece_id=QFFMLCHHLYMQ34BKBGUTXPZBUEOD4MUYM4MMUUSZIIVYPF5JNYRA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.362265206 (503h26m23.233348252s) [noise=true , address=77.90.31.12:4609 , piece_id=BXZJIBBUTJUMS5QXMUO5GJK4FZBLQ5DDBSPQCDLWJSMWBX63K4SQ, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.385788289 (503h26m23.256871324s) [noise=true , address=77.90.31.13:4633 , piece_id=PK62PUCWFHSI4DKUAB5PN4J4LSPG6DMJQRRG6QZPA37ZRGUZYWJA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.471028862 (503h26m23.342111897s) [noise=true , address=77.90.31.21:4910 , piece_id=BLPPM6QRNLPFIGIGBZ7X2IAZNW5JNSVQ6WOF5LAB32IBOYQZABNA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.571884980 (503h26m23.442968015s) [noise=true , address=77.90.31.16:4751 , piece_id=5SV6BRZC2QGJ4NGPJQU6IRUVW56FTFYONW7R5BEICQVK3O3EPGLQ, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.830143060 (503h26m23.701226105s) [noise=true , address=45.91.116.8:4056 , piece_id=ISDYXCRZPW26GEJDM2VZXR6A4XU4PXB2WUQGF3DHXMT44DGADYYQ, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.937329342 (503h26m23.808412387s) [noise=true , address=77.90.31.15:4715 , piece_id=GMHFOJ75XG6LXUZTU5PBBH6F54FYPI4GTZME7FSCUD4SOUWS7TZQ, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.977987703 (503h26m23.849070748s) [noise=true , address=145.79.227.44:4227 , piece_id=CS2BGGIHE5BU6DKYWMEUODVX65Y6XKMN4LTQBNQOVVJN4BJTAEWA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:23.996396166 (503h26m23.867479211s) [noise=true , address=145.79.227.30:4787 , piece_id=F24XWZ57566E7G4X4OXNGI5XM343CH7IIR3QHUKBDJXIEOIZDYBA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:24.024734728 (503h26m23.895817773s) [noise=true , address=145.79.227.40:4057 , piece_id=IK2FBGDQMOSARSMQQTD76YRTMQ6K6AALNOEVIKI43HULEOGN5TUA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: <nil> Dec 26 14:55:24.024758894 (503h26m23.895841939s) [seg_pos={0 0}, upload=441787]: Segment reached optimal threshold of 14 pieces. Dec 26 14:55:24.024969955 (503h26m23.89605299s) [noise=true , address=77.90.31.14:4693 , piece_id=OLPVPOZRUXHJ564V4T7ZJHAGVVOPI7IAV5TQZF72BUJGTQWRMAPA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:24.024978663 (503h26m23.896061698s) [noise=true , address=77.90.31.14:4693 , piece_id=OLPVPOZRUXHJ564V4T7ZJHAGVVOPI7IAV5TQZF72BUJGTQWRMAPA, seg_pos={0 0}, upload=441787]: Long tail cancellation due to optimal threshold Dec 26 14:55:24.025516927 (503h26m23.896599962s) [noise=true , address=45.91.116.12:4238 , piece_id=27VXSSQUTDFRGIIN4K2IEAVWIM7DQWQJ2YPKYJGFDT7IHRTWS5ZA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:24.025524672 (503h26m23.896607707s) [noise=true , address=45.91.116.12:4238 , piece_id=27VXSSQUTDFRGIIN4K2IEAVWIM7DQWQJ2YPKYJGFDT7IHRTWS5ZA, seg_pos={0 0}, upload=441787]: Long tail cancellation due to optimal threshold Dec 26 14:55:24.025635954 (503h26m23.896718989s) [noise=true , address=145.79.227.43:4162 , piece_id=DIT24Q2HJLNDOHCWBEGDTCMFY5QJXXHUKRPPJF2BGJQ2H3SLPWNA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:24.025641064 (503h26m23.896724099s) [noise=true , address=145.79.227.43:4162 , piece_id=DIT24Q2HJLNDOHCWBEGDTCMFY5QJXXHUKRPPJF2BGJQ2H3SLPWNA, seg_pos={0 0}, upload=441787]: Long tail cancellation due to optimal threshold Dec 26 14:55:24.025896971 (503h26m23.896980006s) [noise=true , address=45.91.116.7:4013 , piece_id=VKXKKL6FPFYJXLM567GM2FKHYW7PKVV35YGWENO4GLVD2KR63TDA, seg_pos={0 0}, upload=441787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: read tcp 188.166.225.170:57120->45.91.116.7:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 14:55:24.025904505 (503h26m23.89698755s) [noise=true , address=45.91.116.7:4013 , piece_id=VKXKKL6FPFYJXLM567GM2FKHYW7PKVV35YGWENO4GLVD2KR63TDA, seg_pos={0 0}, upload=441787]: Long tail cancellation due to optimal threshold Dec 26 14:55:24.025919745 (503h26m23.89700279s) [seg_pos={0 0}, upload=441787]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 14:55:24.025990900 (503h26m23.897073935s) [upload=441787]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:55:24.025996771 (503h26m23.897079806s) [upload=441787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:55:24.333961281 (503h26m24.205044316s) [upload=441787]: Done uploading segments... Dec 26 14:55:24.333985157 (503h26m24.205068202s) [upload=441787]: Upload finished. err: <nil> Duration: 1m0.002982272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 14.826305416s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:56:10 UTC) orbiter warning: context deadline exceeded Duration: 14.8263288s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...........X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX.] duration: 6.30914971s trace: https://jaeger.dev.storj.io/trace/28fef441e93b4d1b FAIL (2025-12-26 14:56:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA) context deadline exceeded ======LOGS====== 26 14:55:15.481986730 (503h26m15.353069775s) [upload=441785]: Starting upload Dec 26 14:55:15.482086199 (503h26m15.353169255s) [upload=441785]: Done starting upload Dec 26 14:55:15.482211829 (503h26m15.353294874s) [upload=441785]: Uploading segments... Dec 26 14:55:15.482374409 (503h26m15.353457444s) [upload=441785]: Got next segment. Inline: false Dec 26 14:55:15.482417651 (503h26m15.353500686s) [upload=441785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:55:15.482425215 (503h26m15.35350826s) [upload=441785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:55:15.732804392 (503h26m15.603887447s) [seg_pos={0 0}, upload=441785]: Begin upload segment... Dec 26 14:55:15.732849418 (503h26m15.603932463s) [seg_pos={0 0}, upload=441785]: Done begin upload segment. Dec 26 14:55:15.732884695 (503h26m15.603967741s) [noise=true , address=145.79.227.12:4767 , piece_id=MDTY5FYUYZA5ELZGAAIHDDSNR3BJGAUANQ2FVIZWFWFAT7AU7R2Q, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.732943978 (503h26m15.604027023s) [noise=true , address=145.79.227.37:4566 , piece_id=QLB2F7PN6H3DOEJB644KUS3HNFXZBF232SDACNOEM6BS56JSFZSQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.733486110 (503h26m15.604569155s) [noise=true , address=145.79.227.42:4123 , piece_id=APGOKWFBAUFVCEFVIQ4KOXXYRT5RP4OFDROBG5HSC6ZOXQDPMIDQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.733839894 (503h26m15.604922939s) [noise=true , address=145.79.227.47:4267 , piece_id=RG6ENDWL4VELIUA2JI6BLGQ3ZDN2NMRU7J7ICBS37SLZFXKZXLOA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734163019 (503h26m15.605246054s) [noise=true , address=145.79.227.49:4337 , piece_id=LGUPZGZLOQ2DB6GRHVLA6KLLJOMF5XZKC5ICYPBHF6C6UZ7WO5VQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734457751 (503h26m15.605540796s) [noise=true , address=145.79.227.17:4971 , piece_id=4N5FT4Y77VM7L5J43YIZ6KIYDFZK2VVCP64XLA62X4UONR7Y6HVA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734762611 (503h26m15.605845646s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.735058594 (503h26m15.60614163s) [noise=true , address=45.91.116.11:4120 , piece_id=NSCOJSHLZSVJEQIJJ7IH4KWCDNRM5YUHYQ4TGBOGYXNHJT76VRVQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.735470689 (503h26m15.606553734s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.737056249 (503h26m15.608139294s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.737213398 (503h26m15.608296433s) [noise=true , address=145.79.227.21:4473 , piece_id=U7NGK72WQ5T4DBOZ4GC2Z4BL2OOHY2ZUU7JGCS77CGVTUU6ZNLRA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.738567907 (503h26m15.609650942s) [noise=true , address=145.79.227.19:4078 , piece_id=7LWXRMLUQLKLH3RMLAILN4N6VIKBFH57257RSQVOA2MMC4EFIPAQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.739954838 (503h26m15.611037883s) [noise=true , address=45.91.116.10:4084 , piece_id=2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.741117191 (503h26m15.612200236s) [noise=true , address=45.91.116.12:4227 , piece_id=4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.742502920 (503h26m15.613585955s) [noise=true , address=145.79.227.6:4257 , piece_id=THHD7N43R5OPTGS4A6FW47F2YXXV7TM3RJBIH7VA7WOUFXFQRWKQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.743815189 (503h26m15.614898224s) [noise=true , address=45.91.116.5:4300 , piece_id=PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:17.443210343 (503h26m17.314293388s) [upload=441785]: Next returned nil segment Dec 26 14:55:17.443224791 (503h26m17.314307836s) [upload=441785]: Waiting for error group managing segments... Dec 26 14:55:17.742099611 (503h26m17.613182656s) [noise=true , address=145.79.227.19:4078 , piece_id=7LWXRMLUQLKLH3RMLAILN4N6VIKBFH57257RSQVOA2MMC4EFIPAQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.743202762 (503h26m17.614285807s) [noise=true , address=145.79.227.42:4123 , piece_id=APGOKWFBAUFVCEFVIQ4KOXXYRT5RP4OFDROBG5HSC6ZOXQDPMIDQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.749438656 (503h26m17.620521701s) [noise=true , address=145.79.227.21:4473 , piece_id=U7NGK72WQ5T4DBOZ4GC2Z4BL2OOHY2ZUU7JGCS77CGVTUU6ZNLRA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.753956048 (503h26m17.625039093s) [noise=true , address=145.79.227.47:4267 , piece_id=RG6ENDWL4VELIUA2JI6BLGQ3ZDN2NMRU7J7ICBS37SLZFXKZXLOA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.755334604 (503h26m17.626417639s) [noise=true , address=145.79.227.37:4566 , piece_id=QLB2F7PN6H3DOEJB644KUS3HNFXZBF232SDACNOEM6BS56JSFZSQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.755766065 (503h26m17.6268491s) [noise=true , address=145.79.227.49:4337 , piece_id=LGUPZGZLOQ2DB6GRHVLA6KLLJOMF5XZKC5ICYPBHF6C6UZ7WO5VQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.758197936 (503h26m17.629280971s) [noise=true , address=145.79.227.6:4257 , piece_id=THHD7N43R5OPTGS4A6FW47F2YXXV7TM3RJBIH7VA7WOUFXFQRWKQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.766691289 (503h26m17.637774334s) [noise=true , address=145.79.227.12:4767 , piece_id=MDTY5FYUYZA5ELZGAAIHDDSNR3BJGAUANQ2FVIZWFWFAT7AU7R2Q, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.852019669 (503h26m17.723102714s) [noise=true , address=145.79.227.17:4971 , piece_id=4N5FT4Y77VM7L5J43YIZ6KIYDFZK2VVCP64XLA62X4UONR7Y6HVA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.985792848 (503h26m17.856875883s) [noise=true , address=45.91.116.10:4084 , piece_id=2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.105553042 (503h26m17.976636088s) [noise=true , address=45.91.116.11:4120 , piece_id=NSCOJSHLZSVJEQIJJ7IH4KWCDNRM5YUHYQ4TGBOGYXNHJT76VRVQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.173726066 (503h26m18.044809101s) [noise=true , address=45.91.116.12:4227 , piece_id=4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.189895806 (503h26m18.060978851s) [noise=true , address=45.91.116.5:4300 , piece_id=PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.189920062 (503h26m18.061003097s) [seg_pos={0 0}, upload=441785]: Segment reached optimal threshold of 13 pieces. Dec 26 14:55:18.190151133 (503h26m18.061234168s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection Dec 26 14:55:18.190158887 (503h26m18.061241932s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190364599 (503h26m18.061447644s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:18.190403503 (503h26m18.061486548s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190473005 (503h26m18.06155605s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:18.190481852 (503h26m18.061564887s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190503013 (503h26m18.061586058s) [seg_pos={0 0}, upload=441785]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:55:18.190582794 (503h26m18.061665839s) [upload=441785]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:55:18.190588114 (503h26m18.061671159s) [upload=441785]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:55:18.500579403 (503h26m18.371662448s) [upload=441785]: Done uploading segments... Dec 26 14:55:18.500604862 (503h26m18.371687907s) [upload=441785]: Upload finished. err: <nil> Duration: 6.309189777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...........X....XX.XXX..XXX.X....X.XX.XXX...XX...........XX] duration: 30.003766743s trace: https://jaeger.dev.storj.io/trace/28fef441e93b4d1b FAIL (2025-12-26 14:55:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA) context deadline exceeded ======LOGS====== 26 14:55:15.481986730 (503h26m15.353069775s) [upload=441785]: Starting upload Dec 26 14:55:15.482086199 (503h26m15.353169255s) [upload=441785]: Done starting upload Dec 26 14:55:15.482211829 (503h26m15.353294874s) [upload=441785]: Uploading segments... Dec 26 14:55:15.482374409 (503h26m15.353457444s) [upload=441785]: Got next segment. Inline: false Dec 26 14:55:15.482417651 (503h26m15.353500686s) [upload=441785]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:55:15.482425215 (503h26m15.35350826s) [upload=441785]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:55:15.732804392 (503h26m15.603887447s) [seg_pos={0 0}, upload=441785]: Begin upload segment... Dec 26 14:55:15.732849418 (503h26m15.603932463s) [seg_pos={0 0}, upload=441785]: Done begin upload segment. Dec 26 14:55:15.732884695 (503h26m15.603967741s) [noise=true , address=145.79.227.12:4767 , piece_id=MDTY5FYUYZA5ELZGAAIHDDSNR3BJGAUANQ2FVIZWFWFAT7AU7R2Q, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.732943978 (503h26m15.604027023s) [noise=true , address=145.79.227.37:4566 , piece_id=QLB2F7PN6H3DOEJB644KUS3HNFXZBF232SDACNOEM6BS56JSFZSQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.733486110 (503h26m15.604569155s) [noise=true , address=145.79.227.42:4123 , piece_id=APGOKWFBAUFVCEFVIQ4KOXXYRT5RP4OFDROBG5HSC6ZOXQDPMIDQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.733839894 (503h26m15.604922939s) [noise=true , address=145.79.227.47:4267 , piece_id=RG6ENDWL4VELIUA2JI6BLGQ3ZDN2NMRU7J7ICBS37SLZFXKZXLOA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734163019 (503h26m15.605246054s) [noise=true , address=145.79.227.49:4337 , piece_id=LGUPZGZLOQ2DB6GRHVLA6KLLJOMF5XZKC5ICYPBHF6C6UZ7WO5VQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734457751 (503h26m15.605540796s) [noise=true , address=145.79.227.17:4971 , piece_id=4N5FT4Y77VM7L5J43YIZ6KIYDFZK2VVCP64XLA62X4UONR7Y6HVA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.734762611 (503h26m15.605845646s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.735058594 (503h26m15.60614163s) [noise=true , address=45.91.116.11:4120 , piece_id=NSCOJSHLZSVJEQIJJ7IH4KWCDNRM5YUHYQ4TGBOGYXNHJT76VRVQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.735470689 (503h26m15.606553734s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.737056249 (503h26m15.608139294s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.737213398 (503h26m15.608296433s) [noise=true , address=145.79.227.21:4473 , piece_id=U7NGK72WQ5T4DBOZ4GC2Z4BL2OOHY2ZUU7JGCS77CGVTUU6ZNLRA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.738567907 (503h26m15.609650942s) [noise=true , address=145.79.227.19:4078 , piece_id=7LWXRMLUQLKLH3RMLAILN4N6VIKBFH57257RSQVOA2MMC4EFIPAQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.739954838 (503h26m15.611037883s) [noise=true , address=45.91.116.10:4084 , piece_id=2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.741117191 (503h26m15.612200236s) [noise=true , address=45.91.116.12:4227 , piece_id=4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.742502920 (503h26m15.613585955s) [noise=true , address=145.79.227.6:4257 , piece_id=THHD7N43R5OPTGS4A6FW47F2YXXV7TM3RJBIH7VA7WOUFXFQRWKQ, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:15.743815189 (503h26m15.614898224s) [noise=true , address=45.91.116.5:4300 , piece_id=PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA, seg_pos={0 0}, upload=441785]: Uploading piece... Dec 26 14:55:17.443210343 (503h26m17.314293388s) [upload=441785]: Next returned nil segment Dec 26 14:55:17.443224791 (503h26m17.314307836s) [upload=441785]: Waiting for error group managing segments... Dec 26 14:55:17.742099611 (503h26m17.613182656s) [noise=true , address=145.79.227.19:4078 , piece_id=7LWXRMLUQLKLH3RMLAILN4N6VIKBFH57257RSQVOA2MMC4EFIPAQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.743202762 (503h26m17.614285807s) [noise=true , address=145.79.227.42:4123 , piece_id=APGOKWFBAUFVCEFVIQ4KOXXYRT5RP4OFDROBG5HSC6ZOXQDPMIDQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.749438656 (503h26m17.620521701s) [noise=true , address=145.79.227.21:4473 , piece_id=U7NGK72WQ5T4DBOZ4GC2Z4BL2OOHY2ZUU7JGCS77CGVTUU6ZNLRA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.753956048 (503h26m17.625039093s) [noise=true , address=145.79.227.47:4267 , piece_id=RG6ENDWL4VELIUA2JI6BLGQ3ZDN2NMRU7J7ICBS37SLZFXKZXLOA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.755334604 (503h26m17.626417639s) [noise=true , address=145.79.227.37:4566 , piece_id=QLB2F7PN6H3DOEJB644KUS3HNFXZBF232SDACNOEM6BS56JSFZSQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.755766065 (503h26m17.6268491s) [noise=true , address=145.79.227.49:4337 , piece_id=LGUPZGZLOQ2DB6GRHVLA6KLLJOMF5XZKC5ICYPBHF6C6UZ7WO5VQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.758197936 (503h26m17.629280971s) [noise=true , address=145.79.227.6:4257 , piece_id=THHD7N43R5OPTGS4A6FW47F2YXXV7TM3RJBIH7VA7WOUFXFQRWKQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.766691289 (503h26m17.637774334s) [noise=true , address=145.79.227.12:4767 , piece_id=MDTY5FYUYZA5ELZGAAIHDDSNR3BJGAUANQ2FVIZWFWFAT7AU7R2Q, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.852019669 (503h26m17.723102714s) [noise=true , address=145.79.227.17:4971 , piece_id=4N5FT4Y77VM7L5J43YIZ6KIYDFZK2VVCP64XLA62X4UONR7Y6HVA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:17.985792848 (503h26m17.856875883s) [noise=true , address=45.91.116.10:4084 , piece_id=2OFWGMPWI7EYWS2OFXMTSV5SER5MECGR353HZRS65ODC36K3LORA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.105553042 (503h26m17.976636088s) [noise=true , address=45.91.116.11:4120 , piece_id=NSCOJSHLZSVJEQIJJ7IH4KWCDNRM5YUHYQ4TGBOGYXNHJT76VRVQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.173726066 (503h26m18.044809101s) [noise=true , address=45.91.116.12:4227 , piece_id=4ILPWJSV4R5WELQAT66AXRPJSGQCTY6FZKEVRM3UWQFJ7RRSJ5MQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.189895806 (503h26m18.060978851s) [noise=true , address=45.91.116.5:4300 , piece_id=PRMWZ5X5V4W6BUXDFPGJYFA7QOOGDAO2S6SV27BTWYES3WI7ZPVA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: <nil> Dec 26 14:55:18.189920062 (503h26m18.061003097s) [seg_pos={0 0}, upload=441785]: Segment reached optimal threshold of 13 pieces. Dec 26 14:55:18.190151133 (503h26m18.061234168s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59178->77.90.31.14:4664: use of closed network connection Dec 26 14:55:18.190158887 (503h26m18.061241932s) [noise=true , address=77.90.31.14:4664 , piece_id=MNDMZUL5PMTSLRZMJPUT6EC7GUNIAPC7H3BGRDVPEVUWNXBQMWIA, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190364599 (503h26m18.061447644s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:18.190403503 (503h26m18.061486548s) [noise=true , address=145.79.227.43:4159 , piece_id=JX4ZAWA6FKBEF5IO5ESR2CYLLBNMF4I724AOZRGVWJJ2V6LB7VQA, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190473005 (503h26m18.06155605s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 14:55:18.190481852 (503h26m18.061564887s) [noise=true , address=145.79.227.44:4194 , piece_id=SNOVFUYXBJR73FYDUUHWCISMMBWI4OPT4NMWWE6KLJJLKTTI77RQ, seg_pos={0 0}, upload=441785]: Long tail cancellation due to optimal threshold Dec 26 14:55:18.190503013 (503h26m18.061586058s) [seg_pos={0 0}, upload=441785]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:55:18.190582794 (503h26m18.061665839s) [upload=441785]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:55:18.190588114 (503h26m18.061671159s) [upload=441785]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:55:18.500579403 (503h26m18.371662448s) [upload=441785]: Done uploading segments... Dec 26 14:55:18.500604862 (503h26m18.371687907s) [upload=441785]: Upload finished. err: <nil> Duration: 30.003813743s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXXX] duration: 5.001002147s trace: https://jaeger.dev.storj.io/trace/1ab3a53fbe36f160 FAIL (2025-12-26 14:55:42 UTC) orbiter warning: context deadline exceeded Duration: 5.00103042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 13.360154505s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:55:29 UTC) orbiter warning: context deadline exceeded Duration: 13.36018375s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXXX] duration: 5.001850903s trace: https://jaeger.dev.storj.io/trace/4d7ae578eef143a7 FAIL (2025-12-26 14:55:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001871523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X.X.X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...X] duration: 1m0.003632668s trace: https://jaeger.dev.storj.io/trace/6bfff440deac7ee FAIL (2025-12-26 14:55:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=193.149.164.143:4694 , piece_id=SJTH47E554BHZHAVPXLGMG2USM722H4TCTCHE4QD7HOFDJFK7FWQ, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.895128855 (503h25m20.76621189s) [noise=true , address=77.90.31.10:4545 , piece_id=CXC7VMJGQD5PQJSYBY5EPONPY4ED4MFOPGCFSRZQCCU6ZWCJDN4A, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.895495814 (503h25m20.766578859s) [noise=true , address=77.90.31.18:4826 , piece_id=4BLE427VW6E637RON3FMY7LEPBTO3LUPQVKQ6YNMQKQVV2OZSLLQ, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.895893400 (503h25m20.766976446s) [noise=true , address=64.49.13.140:4515 , piece_id=FNUHCIF53NN6EHY4MQPIGS52I3KKSKRSZ5ASJ3HP5CCKZE7KTAMA, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.896198812 (503h25m20.767281847s) [noise=true , address=145.79.227.43:4169 , piece_id=KJWF2BBWEK7N6DI373OI46KD4YOTFZGPUTVNCKPQRTGSF7ZBFVRQ, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.896506328 (503h25m20.767589363s) [noise=true , address=77.90.31.11:4574 , piece_id=3WD7BAZA3JNRXWXYXEHQFMUFHE3ZW7G5VKBHQU72VKBZZG5RQNSQ, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.896911038 (503h25m20.767994073s) [noise=true , address=77.90.31.21:4889 , piece_id=B4UCLX6R32LATPWRWSQKSMXSD3XW7VXMUOMCVP2FVBLW5NSHFJ3A, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.897731932 (503h25m20.768814977s) [noise=true , address=77.90.31.19:4867 , piece_id=YGA4ERNCVNCQVZMNGIX4XKC4Z6MVPTKJGF2MHFIPCRNFVZYVR4WA, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.898567863 (503h25m20.769650898s) [noise=true , address=77.90.31.14:4695 , piece_id=5CY2XRWNIBX7OZJJZTBVSVWKDBT6XUHSRY723A6WX2T6RWNYMJNQ, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.898780608 (503h25m20.769863643s) [noise=true , address=45.91.116.10:4115 , piece_id=ESC5S3Y6N46YHETGBTAY3WXTSYBII6EBBYFTGX5OCDJ2NNYQ3PWQ, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.900660338 (503h25m20.771743373s) [noise=true , address=145.79.227.40:4076 , piece_id=INVRD5AMAQSPGK5F7Q6DUD4OZFYL5QNRDJUKHV5UCCBOFA24IF4A, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.902190992 (503h25m20.773274037s) [noise=true , address=64.49.13.169:4716 , piece_id=MDWWDQHJX5VJOVDDW3Y5ZXRMFKC5AUGILOCQWLZRCPWNDKYJEDDA, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.903463334 (503h25m20.77454638s) [noise=true , address=77.90.31.12:4622 , piece_id=WKDSV2MZBXWP35KD23NV2KQTRIETSM3TWFHC5ZFTLDP3Z24CO3YA, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.904090098 (503h25m20.775173143s) [noise=true , address=145.79.227.52:4448 , piece_id=GSL6WA7Q5NKSB3EF6B35E5LS2ETHB5YGLIXKNVOTHJD7ZW5IVFGA, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.906787244 (503h25m20.777870279s) [noise=true , address=145.79.227.44:4204 , piece_id=AKTOOVQ545UZ2Q7NT245L4EGO7NKWK5NFVEMNXTENMBYEISOBMRQ, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:20.911620187 (503h25m20.782703232s) [noise=true , address=45.91.116.13:4169 , piece_id=URKTF7XKNSCFP2753TEBCIKCQJSXOTTFZBXFB64KBS4OLOX3SA7A, seg_pos={0 0}, upload=441774]: Uploading piece... Dec 26 14:54:22.210268707 (503h25m22.081351742s) [upload=441774]: Next returned nil segment Dec 26 14:54:22.210282563 (503h25m22.081365598s) [upload=441774]: Waiting for error group managing segments... Dec 26 14:54:22.542570529 (503h25m22.413653564s) [noise=true , address=193.149.164.143:4694 , piece_id=SJTH47E554BHZHAVPXLGMG2USM722H4TCTCHE4QD7HOFDJFK7FWQ, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:22.781487320 (503h25m22.652570355s) [noise=true , address=145.79.227.43:4169 , piece_id=KJWF2BBWEK7N6DI373OI46KD4YOTFZGPUTVNCKPQRTGSF7ZBFVRQ, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:22.838838427 (503h25m22.709921472s) [noise=true , address=145.79.227.52:4448 , piece_id=GSL6WA7Q5NKSB3EF6B35E5LS2ETHB5YGLIXKNVOTHJD7ZW5IVFGA, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:23.067048329 (503h25m22.938131364s) [noise=true , address=77.90.31.21:4889 , piece_id=B4UCLX6R32LATPWRWSQKSMXSD3XW7VXMUOMCVP2FVBLW5NSHFJ3A, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:23.079420969 (503h25m22.950504004s) [noise=true , address=77.90.31.15:4720 , piece_id=RBVA7WEG7KFIGP7XWECYV2LGHHWQXJSQHPMYXF5ZV2L2DQ3MNPBA, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:23.097706569 (503h25m22.968789604s) [noise=true , address=77.90.31.11:4574 , piece_id=3WD7BAZA3JNRXWXYXEHQFMUFHE3ZW7G5VKBHQU72VKBZZG5RQNSQ, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:23.229625325 (503h25m23.10070837s) [noise=true , address=77.90.31.14:4695 , piece_id=5CY2XRWNIBX7OZJJZTBVSVWKDBT6XUHSRY723A6WX2T6RWNYMJNQ, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:23.710140452 (503h25m23.581223487s) [noise=true , address=145.79.227.44:4204 , piece_id=AKTOOVQ545UZ2Q7NT245L4EGO7NKWK5NFVEMNXTENMBYEISOBMRQ, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:23.756489950 (503h25m23.627572995s) [noise=true , address=145.79.227.30:4767 , piece_id=GIWFVAH4GIV4IABL63MV52CKFDNJ5BCJ4N5FYEQ6B6KEGIXHF76A, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:23.931253592 (503h25m23.802336627s) [noise=true , address=45.91.116.13:4169 , piece_id=URKTF7XKNSCFP2753TEBCIKCQJSXOTTFZBXFB64KBS4OLOX3SA7A, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:23.957927784 (503h25m23.829010829s) [noise=true , address=64.49.13.140:4515 , piece_id=FNUHCIF53NN6EHY4MQPIGS52I3KKSKRSZ5ASJ3HP5CCKZE7KTAMA, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:24.070306243 (503h25m23.941389278s) [noise=true , address=77.90.31.12:4622 , piece_id=WKDSV2MZBXWP35KD23NV2KQTRIETSM3TWFHC5ZFTLDP3Z24CO3YA, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:24.131268075 (503h25m24.002351111s) [noise=true , address=77.90.31.10:4545 , piece_id=CXC7VMJGQD5PQJSYBY5EPONPY4ED4MFOPGCFSRZQCCU6ZWCJDN4A, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:24.232916302 (503h25m24.103999337s) [noise=true , address=145.79.227.40:4076 , piece_id=INVRD5AMAQSPGK5F7Q6DUD4OZFYL5QNRDJUKHV5UCCBOFA24IF4A, seg_pos={0 0}, upload=441774]: Done uploading piece. err: <nil> Dec 26 14:54:24.232940038 (503h25m24.104023083s) [seg_pos={0 0}, upload=441774]: Segment reached optimal threshold of 14 pieces. Dec 26 14:54:24.233259185 (503h25m24.10434223s) [noise=true , address=45.91.116.10:4115 , piece_id=ESC5S3Y6N46YHETGBTAY3WXTSYBII6EBBYFTGX5OCDJ2NNYQ3PWQ, seg_pos={0 0}, upload=441774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 26 14:54:24.233271639 (503h25m24.104354704s) [noise=true , address=45.91.116.10:4115 , piece_id=ESC5S3Y6N46YHETGBTAY3WXTSYBII6EBBYFTGX5OCDJ2NNYQ3PWQ, seg_pos={0 0}, upload=441774]: Long tail cancellation due to optimal threshold Dec 26 14:54:24.233288040 (503h25m24.104371085s) [noise=true , address=64.49.13.169:4716 , piece_id=MDWWDQHJX5VJOVDDW3Y5ZXRMFKC5AUGILOCQWLZRCPWNDKYJEDDA, seg_pos={0 0}, upload=441774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:46496->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:46496->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46496->64.49.13.169:4716: use of closed network connection Dec 26 14:54:24.233298490 (503h25m24.104381555s) [noise=true , address=64.49.13.169:4716 , piece_id=MDWWDQHJX5VJOVDDW3Y5ZXRMFKC5AUGILOCQWLZRCPWNDKYJEDDA, seg_pos={0 0}, upload=441774]: Long tail cancellation due to optimal threshold Dec 26 14:54:24.233496928 (503h25m24.104579973s) [noise=true , address=77.90.31.18:4826 , piece_id=4BLE427VW6E637RON3FMY7LEPBTO3LUPQVKQ6YNMQKQVV2OZSLLQ, seg_pos={0 0}, upload=441774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ): protocol: expected piece hash; context canceled; EOF Dec 26 14:54:24.233506035 (503h25m24.10458908s) [noise=true , address=77.90.31.18:4826 , piece_id=4BLE427VW6E637RON3FMY7LEPBTO3LUPQVKQ6YNMQKQVV2OZSLLQ, seg_pos={0 0}, upload=441774]: Long tail cancellation due to optimal threshold Dec 26 14:54:24.233668615 (503h25m24.10475166s) [noise=true , address=77.90.31.19:4867 , piece_id=YGA4ERNCVNCQVZMNGIX4XKC4Z6MVPTKJGF2MHFIPCRNFVZYVR4WA, seg_pos={0 0}, upload=441774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: expected piece hash; context canceled; EOF Dec 26 14:54:24.233677121 (503h25m24.104760156s) [noise=true , address=77.90.31.19:4867 , piece_id=YGA4ERNCVNCQVZMNGIX4XKC4Z6MVPTKJGF2MHFIPCRNFVZYVR4WA, seg_pos={0 0}, upload=441774]: Long tail cancellation due to optimal threshold Dec 26 14:54:24.233695646 (503h25m24.104778691s) [seg_pos={0 0}, upload=441774]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 14:54:24.233770097 (503h25m24.104853133s) [upload=441774]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:54:24.233775879 (503h25m24.104858924s) [upload=441774]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:54:24.532444897 (503h25m24.403527932s) [upload=441774]: Done uploading segments... Dec 26 14:54:24.532463873 (503h25m24.403546909s) [upload=441774]: Upload finished. err: <nil> Duration: 1m0.003646354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X...........X....XX.XXX..XXX.X....X.XX.XXX...XX...........X] duration: 30.002929265s trace: https://jaeger.dev.storj.io/trace/a26a997226cbadb FAIL (2025-12-26 14:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: EHF2RQ3CEV64LPSIILURQROCRJLDBKMF4CQDOMFB7B53QYGOL7EA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: TA7KRGKRYWU6UT7OKM7PK34GC2SFK4RF2CQO7SGVLHXELG5ARCTA) context deadline exceeded ======LOGS====== Dec 26 14:54:09.630525766 (503h25m9.501608811s) [upload=441773]: Starting upload Dec 26 14:54:09.630606540 (503h25m9.501689585s) [upload=441773]: Done starting upload Dec 26 14:54:09.630670963 (503h25m9.501753998s) [upload=441773]: Uploading segments... Dec 26 14:54:09.630827451 (503h25m9.501910486s) [upload=441773]: Got next segment. Inline: false Dec 26 14:54:09.630863650 (503h25m9.501946695s) [upload=441773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:54:09.630868840 (503h25m9.501951875s) [upload=441773]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:54:09.877268791 (503h25m9.748351856s) [seg_pos={0 0}, upload=441773]: Begin upload segment... Dec 26 14:54:09.877327483 (503h25m9.748410538s) [seg_pos={0 0}, upload=441773]: Done begin upload segment. Dec 26 14:54:09.877355787 (503h25m9.748438832s) [noise=true , address=45.91.116.14:4191 , piece_id=TA7KRGKRYWU6UT7OKM7PK34GC2SFK4RF2CQO7SGVLHXELG5ARCTA, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.877365355 (503h25m9.74844841s) [noise=true , address=45.91.116.5:4297 , piece_id=EHF2RQ3CEV64LPSIILURQROCRJLDBKMF4CQDOMFB7B53QYGOL7EA, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.878285366 (503h25m9.749368411s) [noise=true , address=145.79.227.19:4078 , piece_id=SGJHLXEQX77KC65QLCRECA46FKXBET5GXSOWAGZV73UCJU2F634Q, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.878723591 (503h25m9.749806636s) [noise=true , address=145.79.227.40:4051 , piece_id=ZBFODF3HNBFYQUK546K4NMJGAIYROMFSSVULF6IYOZKPPCXW7DFQ, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.879132540 (503h25m9.750215585s) [noise=true , address=77.90.31.21:4878 , piece_id=YVWLM63JBHHIYSSM7JLYJE5IVZPWX7SP2ZF2WUMORTVH2N2RLEUQ, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.879586144 (503h25m9.750669189s) [noise=true , address=145.79.227.11:4731 , piece_id=W5EJAEH6HCGEMPU6NHJR2IBIIGBOSPDGKC6SNXTQ4NPXLMF6K6JQ, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.880016292 (503h25m9.751099337s) [noise=true , address=145.79.227.46:4603 , piece_id=TBSERIUVHIYKWBENB4ABCT6DCGAETMNYFXLH5DFY7QROET7HHHFA, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.880860650 (503h25m9.751943685s) [noise=true , address=145.79.227.45:4230 , piece_id=DIKQZD2YCOZRZLZHWV2YN2PHII3YUUCR7A5ZONZP6OR4CDD2VZ5Q, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.881520907 (503h25m9.752603952s) [noise=true , address=145.79.227.51:4411 , piece_id=SCNU2THCB2FWWM67KGVX6PYPPA2QK35QODRFY5LHDW3IW3PPCUZA, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.882191193 (503h25m9.753274228s) [noise=true , address=45.91.116.13:4155 , piece_id=Z56M2I342TDYRMMG3APBX4IX3PO5HEC3PBNYHGPND7IAFKSF6L3A, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.883284075 (503h25m9.75436712s) [noise=true , address=145.79.227.15:4876 , piece_id=EHNGP3S5DYUHR2XMBE5L5GQX3CQBUYXJ2G5NMMALE7PCU4UIEV3Q, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.883368545 (503h25m9.75445159s) [noise=true , address=45.91.116.11:4119 , piece_id=HBSDBHSB2MC2OQSSBRWBZXH446N42A43MWOJREZUWLKKPEY374EQ, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.884397796 (503h25m9.755480831s) [noise=true , address=145.79.227.7:4294 , piece_id=PKR5SSFDKEJNU56YYKGCKHEVWLOSXPH5IHPCURZUTKBRYPBIWCAA, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.885381458 (503h25m9.756464504s) [noise=true , address=145.79.227.47:4266 , piece_id=SN2NDFBZ6KKRIJK3SWUV5N7UTG7B44OZQE7NQK3GPPER44HFDVRA, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.887127082 (503h25m9.758210127s) [noise=true , address=145.79.227.42:4123 , piece_id=SZQWP7JMTZ3D76EJAAFA2ODQORMKLFDCVE472HSU3UXAPVBZOBHA, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:09.888725366 (503h25m9.759808411s) [noise=true , address=145.79.227.52:4447 , piece_id=ETQHGABYAWDNEDHOMHMPIW7U2UCIWUS77CUMJM6U247IEJ4O65OA, seg_pos={0 0}, upload=441773]: Uploading piece... Dec 26 14:54:11.580073385 (503h25m11.45115643s) [upload=441773]: Next returned nil segment Dec 26 14:54:11.580099866 (503h25m11.451182911s) [upload=441773]: Waiting for error group managing segments... Dec 26 14:54:11.882177526 (503h25m11.753260561s) [noise=true , address=145.79.227.45:4230 , piece_id=DIKQZD2YCOZRZLZHWV2YN2PHII3YUUCR7A5ZONZP6OR4CDD2VZ5Q, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.887998150 (503h25m11.759081195s) [noise=true , address=145.79.227.51:4411 , piece_id=SCNU2THCB2FWWM67KGVX6PYPPA2QK35QODRFY5LHDW3IW3PPCUZA, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.888080788 (503h25m11.759163833s) [noise=true , address=145.79.227.15:4876 , piece_id=EHNGP3S5DYUHR2XMBE5L5GQX3CQBUYXJ2G5NMMALE7PCU4UIEV3Q, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.891126246 (503h25m11.762209281s) [noise=true , address=145.79.227.11:4731 , piece_id=W5EJAEH6HCGEMPU6NHJR2IBIIGBOSPDGKC6SNXTQ4NPXLMF6K6JQ, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.904952484 (503h25m11.776035529s) [noise=true , address=145.79.227.46:4603 , piece_id=TBSERIUVHIYKWBENB4ABCT6DCGAETMNYFXLH5DFY7QROET7HHHFA, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.914951735 (503h25m11.78603478s) [noise=true , address=145.79.227.52:4447 , piece_id=ETQHGABYAWDNEDHOMHMPIW7U2UCIWUS77CUMJM6U247IEJ4O65OA, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.915376583 (503h25m11.786459628s) [noise=true , address=145.79.227.7:4294 , piece_id=PKR5SSFDKEJNU56YYKGCKHEVWLOSXPH5IHPCURZUTKBRYPBIWCAA, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.916561480 (503h25m11.787644525s) [noise=true , address=145.79.227.42:4123 , piece_id=SZQWP7JMTZ3D76EJAAFA2ODQORMKLFDCVE472HSU3UXAPVBZOBHA, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.919078592 (503h25m11.790161627s) [noise=true , address=145.79.227.19:4078 , piece_id=SGJHLXEQX77KC65QLCRECA46FKXBET5GXSOWAGZV73UCJU2F634Q, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:11.926803562 (503h25m11.797886607s) [noise=true , address=145.79.227.47:4266 , piece_id=SN2NDFBZ6KKRIJK3SWUV5N7UTG7B44OZQE7NQK3GPPER44HFDVRA, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:12.121664402 (503h25m11.992747447s) [noise=true , address=45.91.116.14:4191 , piece_id=TA7KRGKRYWU6UT7OKM7PK34GC2SFK4RF2CQO7SGVLHXELG5ARCTA, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:12.365488787 (503h25m12.236571833s) [noise=true , address=45.91.116.5:4297 , piece_id=EHF2RQ3CEV64LPSIILURQROCRJLDBKMF4CQDOMFB7B53QYGOL7EA, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:13.010017867 (503h25m12.881100912s) [noise=true , address=45.91.116.11:4119 , piece_id=HBSDBHSB2MC2OQSSBRWBZXH446N42A43MWOJREZUWLKKPEY374EQ, seg_pos={0 0}, upload=441773]: Done uploading piece. err: <nil> Dec 26 14:54:13.010042704 (503h25m12.881125739s) [seg_pos={0 0}, upload=441773]: Segment reached optimal threshold of 13 pieces. Dec 26 14:54:13.010298661 (503h25m12.881381697s) [noise=true , address=77.90.31.21:4878 , piece_id=YVWLM63JBHHIYSSM7JLYJE5IVZPWX7SP2ZF2WUMORTVH2N2RLEUQ, seg_pos={0 0}, upload=441773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 14:54:13.010312729 (503h25m12.881395764s) [noise=true , address=77.90.31.21:4878 , piece_id=YVWLM63JBHHIYSSM7JLYJE5IVZPWX7SP2ZF2WUMORTVH2N2RLEUQ, seg_pos={0 0}, upload=441773]: Long tail cancellation due to optimal threshold Dec 26 14:54:13.010356061 (503h25m12.881439106s) [noise=true , address=45.91.116.13:4155 , piece_id=Z56M2I342TDYRMMG3APBX4IX3PO5HEC3PBNYHGPND7IAFKSF6L3A, seg_pos={0 0}, upload=441773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 14:54:13.010365539 (503h25m12.881448584s) [noise=true , address=45.91.116.13:4155 , piece_id=Z56M2I342TDYRMMG3APBX4IX3PO5HEC3PBNYHGPND7IAFKSF6L3A, seg_pos={0 0}, upload=441773]: Long tail cancellation due to optimal threshold Dec 26 14:54:13.010492080 (503h25m12.881575115s) [noise=true , address=145.79.227.40:4051 , piece_id=ZBFODF3HNBFYQUK546K4NMJGAIYROMFSSVULF6IYOZKPPCXW7DFQ, seg_pos={0 0}, upload=441773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 14:54:13.010500115 (503h25m12.88158316s) [noise=true , address=145.79.227.40:4051 , piece_id=ZBFODF3HNBFYQUK546K4NMJGAIYROMFSSVULF6IYOZKPPCXW7DFQ, seg_pos={0 0}, upload=441773]: Long tail cancellation due to optimal threshold Dec 26 14:54:13.010532778 (503h25m12.881615823s) [seg_pos={0 0}, upload=441773]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:54:13.010613701 (503h25m12.881696736s) [upload=441773]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:54:13.010620485 (503h25m12.88170353s) [upload=441773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:54:13.307136003 (503h25m13.178219048s) [upload=441773]: Done uploading segments... Dec 26 14:54:13.307164207 (503h25m13.178247252s) [upload=441773]: Upload finished. err: <nil> Duration: 30.002950146s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 9.548971777s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:54:34 UTC) orbiter warning: context deadline exceeded Duration: 9.549080044s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X.X.X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X...] duration: 50.244674377s trace: https://jaeger.dev.storj.io/trace/1e4c2b957fc67774 FAIL (2025-12-26 14:54:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0:39.262204371 (503h21m39.133287416s) [noise=true , address=77.90.31.12:4600 , piece_id=5LQFCLXWXDTUDM36XB5D7ICZWV5GJRKSINXW2FLSACOXQ5Q5ZWSA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.262855241 (503h21m39.133938286s) [noise=true , address=45.91.116.13:4185 , piece_id=TDUSE3NAFPNHSXOCH7ZAD5CD4IS3YGHXK3LUPDIXX5MJRADPZDDA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.263292013 (503h21m39.134375058s) [noise=true , address=77.90.31.15:4728 , piece_id=OH4ZWYE3HMNTZUKGZV6XGEVE67CZKMPTIYKURMSRZVXK3MH563JQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.263675533 (503h21m39.134758578s) [noise=true , address=145.79.227.43:4186 , piece_id=V7JG76EKIRYWEPO4NKV4UDZSK3T6QPXT7J5EIOSVZCTOIQT33QSQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.264092577 (503h21m39.135175622s) [noise=true , address=145.79.227.44:4205 , piece_id=CPCREAYO557KAIBMBKEIZNWZ25YDPRSXGJMXOASPEKRRGPKFF3NA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.264509992 (503h21m39.135593037s) [noise=true , address=64.49.13.27:4960 , piece_id=5APCLJD37IXY34RRJQ2FMLJZMZZ6VY6ZMZJIWJWT4BHTQ3CXAAKA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.264909252 (503h21m39.135992297s) [noise=true , address=77.90.31.14:4696 , piece_id=457RLBGIVHNB5IGVXG3N4DUENNJ5M4RU7L3TD4DJNXCS3WHUIO4Q, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.265423562 (503h21m39.136506607s) [noise=true , address=145.79.227.49:4354 , piece_id=QBVA6JZHUEZDPXFVTRWWTBYDWRW3ACNPHUHO5RJ72OAKGQKCRR6Q, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.265960373 (503h21m39.137043418s) [noise=true , address=77.90.31.16:4740 , piece_id=BWBQ26GVCZJFH6MIQWLJPSRYLNE46FF7ISIFUUNWKEVXQK7HWCBA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.267862295 (503h21m39.13894534s) [noise=true , address=64.49.13.140:4514 , piece_id=YW57E2EMOFNOOCCEU32F2MC6EXWNKNWKCQTDYD4E3PIDUMDWRKQQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.269598271 (503h21m39.140681316s) [noise=true , address=145.79.227.40:4065 , piece_id=3K537P64SIWC7E7YUF23HFLLX63MC4QOXHSBLCSTAYMNDOAPV2OA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.271282849 (503h21m39.142365894s) [noise=true , address=145.79.227.34:4938 , piece_id=BXQ3QYCKQU4HOIEHKKXKULRQUG6DQ7GMSRF7RHCCSPRX2HQ7H5IQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.272843541 (503h21m39.143926576s) [noise=true , address=145.79.227.33:4902 , piece_id=BXWS7VS4LAZYLVSUVD73NADYN5BBB4BZVBFGCIRFKSZCAGRSBHRA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.277089807 (503h21m39.148172872s) [noise=true , address=77.90.31.11:4580 , piece_id=FDDSE7DP6D5FFB4DKMIKFXUYWFGHD7ZNHA3336PKTSAHVNFSVD2A, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.280028963 (503h21m39.151111998s) [noise=true , address=45.91.116.10:4103 , piece_id=HSKW3JAF35RJ3WQORTCUAR5NWLQWRYHR3WNWXGJP3N65KO52GMZA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.283107425 (503h21m39.15419047s) [noise=true , address=77.90.31.21:4907 , piece_id=RVW3DNWB4TRG7AUNGSZQPPUB2JBBARSEKEUIEK644CZH3FY3NIWQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.287949155 (503h21m39.15903219s) [noise=true , address=77.90.31.18:4827 , piece_id=NR7KYCLNPRBMLOBJTHTXP5U7EGWHUXDJVBJPYEOX3FMFXPQC7UAQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:40.561704914 (503h21m40.432787949s) [upload=441726]: Next returned nil segment Dec 26 14:50:40.561732226 (503h21m40.432815271s) [upload=441726]: Waiting for error group managing segments... Dec 26 14:50:40.904528712 (503h21m40.775611758s) [noise=true , address=64.49.13.27:4960 , piece_id=5APCLJD37IXY34RRJQ2FMLJZMZZ6VY6ZMZJIWJWT4BHTQ3CXAAKA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:40.920992613 (503h21m40.792075648s) [noise=true , address=64.49.13.140:4514 , piece_id=YW57E2EMOFNOOCCEU32F2MC6EXWNKNWKCQTDYD4E3PIDUMDWRKQQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.188523771 (503h21m41.059606806s) [noise=true , address=145.79.227.49:4354 , piece_id=QBVA6JZHUEZDPXFVTRWWTBYDWRW3ACNPHUHO5RJ72OAKGQKCRR6Q, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.442234051 (503h21m41.313317096s) [noise=true , address=77.90.31.21:4907 , piece_id=RVW3DNWB4TRG7AUNGSZQPPUB2JBBARSEKEUIEK644CZH3FY3NIWQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.571405826 (503h21m41.442488861s) [noise=true , address=77.90.31.15:4728 , piece_id=OH4ZWYE3HMNTZUKGZV6XGEVE67CZKMPTIYKURMSRZVXK3MH563JQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.740912368 (503h21m41.611995553s) [noise=true , address=45.91.116.10:4103 , piece_id=HSKW3JAF35RJ3WQORTCUAR5NWLQWRYHR3WNWXGJP3N65KO52GMZA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.771818218 (503h21m41.642901263s) [noise=true , address=77.90.31.12:4600 , piece_id=5LQFCLXWXDTUDM36XB5D7ICZWV5GJRKSINXW2FLSACOXQ5Q5ZWSA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.990111908 (503h21m41.861194953s) [noise=true , address=145.79.227.44:4205 , piece_id=CPCREAYO557KAIBMBKEIZNWZ25YDPRSXGJMXOASPEKRRGPKFF3NA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.031854774 (503h21m41.90293782s) [noise=true , address=145.79.227.34:4938 , piece_id=BXQ3QYCKQU4HOIEHKKXKULRQUG6DQ7GMSRF7RHCCSPRX2HQ7H5IQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.061338116 (503h21m41.932421161s) [noise=true , address=145.79.227.30:4796 , piece_id=PKM4GZZA5Y7VUAGVURSLUEY72GHKLBVFD5YD3CIYKVRRI45YYT6Q, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.162981974 (503h21m42.03406502s) [noise=true , address=145.79.227.43:4186 , piece_id=V7JG76EKIRYWEPO4NKV4UDZSK3T6QPXT7J5EIOSVZCTOIQT33QSQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.194032500 (503h21m42.065115535s) [noise=true , address=145.79.227.33:4902 , piece_id=BXWS7VS4LAZYLVSUVD73NADYN5BBB4BZVBFGCIRFKSZCAGRSBHRA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.288517897 (503h21m42.159600942s) [noise=true , address=77.90.31.16:4740 , piece_id=BWBQ26GVCZJFH6MIQWLJPSRYLNE46FF7ISIFUUNWKEVXQK7HWCBA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.577006872 (503h21m42.448089917s) [noise=true , address=145.79.227.40:4065 , piece_id=3K537P64SIWC7E7YUF23HFLLX63MC4QOXHSBLCSTAYMNDOAPV2OA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.577039193 (503h21m42.448122228s) [seg_pos={0 0}, upload=441726]: Segment reached optimal threshold of 14 pieces. Dec 26 14:50:42.577273911 (503h21m42.448356956s) [noise=true , address=45.91.116.13:4185 , piece_id=TDUSE3NAFPNHSXOCH7ZAD5CD4IS3YGHXK3LUPDIXX5MJRADPZDDA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124VEt9UUm1xfNyKx1sYngqg4GnKcp6S9wtPWMairzjbQ2PpWAq): protocol: expected piece hash; context canceled; EOF Dec 26 14:50:42.577280623 (503h21m42.448363658s) [noise=true , address=45.91.116.13:4185 , piece_id=TDUSE3NAFPNHSXOCH7ZAD5CD4IS3YGHXK3LUPDIXX5MJRADPZDDA, seg_pos={0 0}, upload=441726]: Long tail cancellation due to optimal threshold Dec 26 14:50:42.577365735 (503h21m42.44844877s) [noise=true , address=77.90.31.18:4827 , piece_id=NR7KYCLNPRBMLOBJTHTXP5U7EGWHUXDJVBJPYEOX3FMFXPQC7UAQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 26 14:50:42.577382547 (503h21m42.448465612s) [noise=true , address=77.90.31.18:4827 , piece_id=NR7KYCLNPRBMLOBJTHTXP5U7EGWHUXDJVBJPYEOX3FMFXPQC7UAQ, seg_pos={0 0}, upload=441726]: Long tail cancellation due to optimal threshold Dec 26 14:50:42.577539466 (503h21m42.448622511s) [noise=true , address=77.90.31.14:4696 , piece_id=457RLBGIVHNB5IGVXG3N4DUENNJ5M4RU7L3TD4DJNXCS3WHUIO4Q, seg_pos={0 0}, upload=441726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK): protocol: expected piece hash; context canceled; EOF Dec 26 14:50:42.577547691 (503h21m42.448630726s) [noise=true , address=77.90.31.14:4696 , piece_id=457RLBGIVHNB5IGVXG3N4DUENNJ5M4RU7L3TD4DJNXCS3WHUIO4Q, seg_pos={0 0}, upload=441726]: Long tail cancellation due to optimal threshold Dec 26 14:50:42.577569493 (503h21m42.448652538s) [noise=true , address=77.90.31.11:4580 , piece_id=FDDSE7DP6D5FFB4DKMIKFXUYWFGHD7ZNHA3336PKTSAHVNFSVD2A, seg_pos={0 0}, upload=441726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D): protocol: expected piece hash; context canceled; EOF Dec 26 14:50:42.577578991 (503h21m42.448662036s) [noise=true , address=77.90.31.11:4580 , piece_id=FDDSE7DP6D5FFB4DKMIKFXUYWFGHD7ZNHA3336PKTSAHVNFSVD2A, seg_pos={0 0}, upload=441726]: Long tail cancellation due to optimal threshold Dec 26 14:50:42.577610290 (503h21m42.448693325s) [seg_pos={0 0}, upload=441726]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 14:50:42.577720600 (503h21m42.448803655s) [upload=441726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:50:42.577727845 (503h21m42.44881089s) [upload=441726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:50:42.880809497 (503h21m42.751892542s) [upload=441726]: Done uploading segments... Dec 26 14:50:42.880834295 (503h21m42.75191734s) [upload=441726]: Upload finished. err: <nil> Duration: 50.244691571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XXX] duration: 5.001971543s trace: https://jaeger.dev.storj.io/trace/3ad55d93c4168e53 FAIL (2025-12-26 14:54:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001981182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 8.825384496s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:54:01 UTC) orbiter warning: context deadline exceeded Duration: 8.825416256s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..XX] duration: 5.001375518s trace: https://jaeger.dev.storj.io/trace/6ea0cb086f29cda9 FAIL (2025-12-26 14:53:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001397641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X...........X....XX.XXX..XXX.X....X.XX.XXX...XX...........] duration: 6.340678782s trace: https://jaeger.dev.storj.io/trace/7de0b83678515694 FAIL (2025-12-26 14:53:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ) context deadline exceeded ======LOGS====== egments... Dec 26 14:45:35.590660048 (503h16m35.461743103s) [upload=441652]: Got next segment. Inline: false Dec 26 14:45:35.590699122 (503h16m35.461782177s) [upload=441652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:45:35.590706647 (503h16m35.461789712s) [upload=441652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:45:35.842738340 (503h16m35.713821375s) [seg_pos={0 0}, upload=441652]: Begin upload segment... Dec 26 14:45:35.842786903 (503h16m35.713869938s) [seg_pos={0 0}, upload=441652]: Done begin upload segment. Dec 26 14:45:35.842815106 (503h16m35.713898151s) [noise=true , address=45.91.116.11:4120 , piece_id=JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.842894498 (503h16m35.713977533s) [noise=true , address=145.79.227.51:4409 , piece_id=7PU67WZFLRB7VKQ6B6UNYAA3GH5GPKDPFQZ2GZXL2MJLYC7QQKXA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843349324 (503h16m35.714432359s) [noise=true , address=145.79.227.49:4338 , piece_id=JHBKKSKNDQ4LG35IK3GNOLLFFPEQ3AAADPO66TJGIDZFBMKUTWOA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843718797 (503h16m35.714801832s) [noise=true , address=145.79.227.42:4121 , piece_id=HNITZLSZHAARISPYGXZPEKEKIP2FY43PJABA5TM7HMQ5M2YG5K2A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843994342 (503h16m35.715077377s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.844554508 (503h16m35.715637554s) [noise=true , address=145.79.227.14:4840 , piece_id=BTQ2VYMNB2H5JFP5AQNGKYSFMXTDKFQDT2WCOTVNHZTHAWS3GJEQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.844855001 (503h16m35.715938046s) [noise=true , address=145.79.227.5:4222 , piece_id=ZRB7VEUYC6WMDITAVXIZ6IUHEBKR5TKBP6QLY3TCLDWOXFAUVYNQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845090329 (503h16m35.716173364s) [noise=true , address=145.79.227.13:4803 , piece_id=CT3JJNSTE4ACBQYIIDFTEIDA3SHNCM2RZHVCQY62H3JWSB52IU5A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845204667 (503h16m35.716287712s) [noise=true , address=45.91.116.13:4156 , piece_id=55AO4MBH3XO5IDMOWW2U2UOR6VP3E2YTA5GBUZDFJ33X3MOQ7R2A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845572898 (503h16m35.716655933s) [noise=true , address=77.90.31.13:4629 , piece_id=6NVZXZWVYHEZ4RFXV6SX5QK4QQA3AHYI5WW5TYTU7CRIK5NH7MEQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845635296 (503h16m35.716718341s) [noise=true , address=145.79.227.6:4257 , piece_id=3667CXHPX6A7QSVZFWXWX554MAKS6XMEUIMT3G3E4YGKO6IVRMBA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846069223 (503h16m35.717152268s) [noise=true , address=145.79.227.22:4511 , piece_id=WTBEJE2XO33SIYQY2544KRMEYWGVK72FJFQJM4KY3G6XEA6MY3VA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846439247 (503h16m35.717522282s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846783553 (503h16m35.717866598s) [noise=true , address=45.91.116.6:4973 , piece_id=GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.847545484 (503h16m35.718628529s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.847695220 (503h16m35.718778255s) [noise=true , address=145.79.227.7:4294 , piece_id=HWHDWATE46DSLHWD5KU3K7GVIKKYG74G2F4OHP2FHPMGA6MXN7AQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:37.558956427 (503h16m37.430039462s) [upload=441652]: Next returned nil segment Dec 26 14:45:37.558970595 (503h16m37.43005363s) [upload=441652]: Waiting for error group managing segments... Dec 26 14:45:37.846688381 (503h16m37.717771416s) [noise=true , address=145.79.227.13:4803 , piece_id=CT3JJNSTE4ACBQYIIDFTEIDA3SHNCM2RZHVCQY62H3JWSB52IU5A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.847501870 (503h16m37.718584935s) [noise=true , address=145.79.227.5:4222 , piece_id=ZRB7VEUYC6WMDITAVXIZ6IUHEBKR5TKBP6QLY3TCLDWOXFAUVYNQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.848386425 (503h16m37.71946947s) [noise=true , address=145.79.227.51:4409 , piece_id=7PU67WZFLRB7VKQ6B6UNYAA3GH5GPKDPFQZ2GZXL2MJLYC7QQKXA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.849885419 (503h16m37.720968454s) [noise=true , address=145.79.227.42:4121 , piece_id=HNITZLSZHAARISPYGXZPEKEKIP2FY43PJABA5TM7HMQ5M2YG5K2A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.851684776 (503h16m37.722767821s) [noise=true , address=145.79.227.7:4294 , piece_id=HWHDWATE46DSLHWD5KU3K7GVIKKYG74G2F4OHP2FHPMGA6MXN7AQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.861840313 (503h16m37.732923378s) [noise=true , address=145.79.227.49:4338 , piece_id=JHBKKSKNDQ4LG35IK3GNOLLFFPEQ3AAADPO66TJGIDZFBMKUTWOA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.864286882 (503h16m37.735369917s) [noise=true , address=145.79.227.14:4840 , piece_id=BTQ2VYMNB2H5JFP5AQNGKYSFMXTDKFQDT2WCOTVNHZTHAWS3GJEQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.879425859 (503h16m37.750508905s) [noise=true , address=145.79.227.22:4511 , piece_id=WTBEJE2XO33SIYQY2544KRMEYWGVK72FJFQJM4KY3G6XEA6MY3VA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.881350424 (503h16m37.752433459s) [noise=true , address=145.79.227.6:4257 , piece_id=3667CXHPX6A7QSVZFWXWX554MAKS6XMEUIMT3G3E4YGKO6IVRMBA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.052408239 (503h16m37.923491284s) [noise=true , address=77.90.31.13:4629 , piece_id=6NVZXZWVYHEZ4RFXV6SX5QK4QQA3AHYI5WW5TYTU7CRIK5NH7MEQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.088339709 (503h16m37.959422744s) [noise=true , address=45.91.116.13:4156 , piece_id=55AO4MBH3XO5IDMOWW2U2UOR6VP3E2YTA5GBUZDFJ33X3MOQ7R2A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.092624368 (503h16m37.963707433s) [noise=true , address=45.91.116.11:4120 , piece_id=JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.203970160 (503h16m38.075053195s) [noise=true , address=45.91.116.6:4973 , piece_id=GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.203989287 (503h16m38.075072332s) [seg_pos={0 0}, upload=441652]: Segment reached optimal threshold of 13 pieces. Dec 26 14:45:38.204144872 (503h16m38.075227917s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection Dec 26 14:45:38.204151044 (503h16m38.075234099s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204230095 (503h16m38.07531314s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection Dec 26 14:45:38.204251986 (503h16m38.075335021s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204453320 (503h16m38.075536355s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 14:45:38.204461515 (503h16m38.07554456s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204479620 (503h16m38.075562655s) [seg_pos={0 0}, upload=441652]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:45:38.204555304 (503h16m38.075638339s) [upload=441652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:45:38.204560384 (503h16m38.075643419s) [upload=441652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:45:38.504709632 (503h16m38.375792677s) [upload=441652]: Done uploading segments... Dec 26 14:45:38.504726345 (503h16m38.37580938s) [upload=441652]: Upload finished. err: <nil> Duration: 6.340706253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X.X.X....X.....XXX.XX.XX.....X...XX.X..XX..X.X..X..X.X..] duration: 55.750253791s trace: https://jaeger.dev.storj.io/trace/1e4c2b957fc67774 FAIL (2025-12-26 14:53:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0:39.262204371 (503h21m39.133287416s) [noise=true , address=77.90.31.12:4600 , piece_id=5LQFCLXWXDTUDM36XB5D7ICZWV5GJRKSINXW2FLSACOXQ5Q5ZWSA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.262855241 (503h21m39.133938286s) [noise=true , address=45.91.116.13:4185 , piece_id=TDUSE3NAFPNHSXOCH7ZAD5CD4IS3YGHXK3LUPDIXX5MJRADPZDDA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.263292013 (503h21m39.134375058s) [noise=true , address=77.90.31.15:4728 , piece_id=OH4ZWYE3HMNTZUKGZV6XGEVE67CZKMPTIYKURMSRZVXK3MH563JQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.263675533 (503h21m39.134758578s) [noise=true , address=145.79.227.43:4186 , piece_id=V7JG76EKIRYWEPO4NKV4UDZSK3T6QPXT7J5EIOSVZCTOIQT33QSQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.264092577 (503h21m39.135175622s) [noise=true , address=145.79.227.44:4205 , piece_id=CPCREAYO557KAIBMBKEIZNWZ25YDPRSXGJMXOASPEKRRGPKFF3NA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.264509992 (503h21m39.135593037s) [noise=true , address=64.49.13.27:4960 , piece_id=5APCLJD37IXY34RRJQ2FMLJZMZZ6VY6ZMZJIWJWT4BHTQ3CXAAKA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.264909252 (503h21m39.135992297s) [noise=true , address=77.90.31.14:4696 , piece_id=457RLBGIVHNB5IGVXG3N4DUENNJ5M4RU7L3TD4DJNXCS3WHUIO4Q, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.265423562 (503h21m39.136506607s) [noise=true , address=145.79.227.49:4354 , piece_id=QBVA6JZHUEZDPXFVTRWWTBYDWRW3ACNPHUHO5RJ72OAKGQKCRR6Q, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.265960373 (503h21m39.137043418s) [noise=true , address=77.90.31.16:4740 , piece_id=BWBQ26GVCZJFH6MIQWLJPSRYLNE46FF7ISIFUUNWKEVXQK7HWCBA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.267862295 (503h21m39.13894534s) [noise=true , address=64.49.13.140:4514 , piece_id=YW57E2EMOFNOOCCEU32F2MC6EXWNKNWKCQTDYD4E3PIDUMDWRKQQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.269598271 (503h21m39.140681316s) [noise=true , address=145.79.227.40:4065 , piece_id=3K537P64SIWC7E7YUF23HFLLX63MC4QOXHSBLCSTAYMNDOAPV2OA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.271282849 (503h21m39.142365894s) [noise=true , address=145.79.227.34:4938 , piece_id=BXQ3QYCKQU4HOIEHKKXKULRQUG6DQ7GMSRF7RHCCSPRX2HQ7H5IQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.272843541 (503h21m39.143926576s) [noise=true , address=145.79.227.33:4902 , piece_id=BXWS7VS4LAZYLVSUVD73NADYN5BBB4BZVBFGCIRFKSZCAGRSBHRA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.277089807 (503h21m39.148172872s) [noise=true , address=77.90.31.11:4580 , piece_id=FDDSE7DP6D5FFB4DKMIKFXUYWFGHD7ZNHA3336PKTSAHVNFSVD2A, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.280028963 (503h21m39.151111998s) [noise=true , address=45.91.116.10:4103 , piece_id=HSKW3JAF35RJ3WQORTCUAR5NWLQWRYHR3WNWXGJP3N65KO52GMZA, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.283107425 (503h21m39.15419047s) [noise=true , address=77.90.31.21:4907 , piece_id=RVW3DNWB4TRG7AUNGSZQPPUB2JBBARSEKEUIEK644CZH3FY3NIWQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:39.287949155 (503h21m39.15903219s) [noise=true , address=77.90.31.18:4827 , piece_id=NR7KYCLNPRBMLOBJTHTXP5U7EGWHUXDJVBJPYEOX3FMFXPQC7UAQ, seg_pos={0 0}, upload=441726]: Uploading piece... Dec 26 14:50:40.561704914 (503h21m40.432787949s) [upload=441726]: Next returned nil segment Dec 26 14:50:40.561732226 (503h21m40.432815271s) [upload=441726]: Waiting for error group managing segments... Dec 26 14:50:40.904528712 (503h21m40.775611758s) [noise=true , address=64.49.13.27:4960 , piece_id=5APCLJD37IXY34RRJQ2FMLJZMZZ6VY6ZMZJIWJWT4BHTQ3CXAAKA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:40.920992613 (503h21m40.792075648s) [noise=true , address=64.49.13.140:4514 , piece_id=YW57E2EMOFNOOCCEU32F2MC6EXWNKNWKCQTDYD4E3PIDUMDWRKQQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.188523771 (503h21m41.059606806s) [noise=true , address=145.79.227.49:4354 , piece_id=QBVA6JZHUEZDPXFVTRWWTBYDWRW3ACNPHUHO5RJ72OAKGQKCRR6Q, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.442234051 (503h21m41.313317096s) [noise=true , address=77.90.31.21:4907 , piece_id=RVW3DNWB4TRG7AUNGSZQPPUB2JBBARSEKEUIEK644CZH3FY3NIWQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.571405826 (503h21m41.442488861s) [noise=true , address=77.90.31.15:4728 , piece_id=OH4ZWYE3HMNTZUKGZV6XGEVE67CZKMPTIYKURMSRZVXK3MH563JQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.740912368 (503h21m41.611995553s) [noise=true , address=45.91.116.10:4103 , piece_id=HSKW3JAF35RJ3WQORTCUAR5NWLQWRYHR3WNWXGJP3N65KO52GMZA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.771818218 (503h21m41.642901263s) [noise=true , address=77.90.31.12:4600 , piece_id=5LQFCLXWXDTUDM36XB5D7ICZWV5GJRKSINXW2FLSACOXQ5Q5ZWSA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:41.990111908 (503h21m41.861194953s) [noise=true , address=145.79.227.44:4205 , piece_id=CPCREAYO557KAIBMBKEIZNWZ25YDPRSXGJMXOASPEKRRGPKFF3NA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.031854774 (503h21m41.90293782s) [noise=true , address=145.79.227.34:4938 , piece_id=BXQ3QYCKQU4HOIEHKKXKULRQUG6DQ7GMSRF7RHCCSPRX2HQ7H5IQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.061338116 (503h21m41.932421161s) [noise=true , address=145.79.227.30:4796 , piece_id=PKM4GZZA5Y7VUAGVURSLUEY72GHKLBVFD5YD3CIYKVRRI45YYT6Q, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.162981974 (503h21m42.03406502s) [noise=true , address=145.79.227.43:4186 , piece_id=V7JG76EKIRYWEPO4NKV4UDZSK3T6QPXT7J5EIOSVZCTOIQT33QSQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.194032500 (503h21m42.065115535s) [noise=true , address=145.79.227.33:4902 , piece_id=BXWS7VS4LAZYLVSUVD73NADYN5BBB4BZVBFGCIRFKSZCAGRSBHRA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.288517897 (503h21m42.159600942s) [noise=true , address=77.90.31.16:4740 , piece_id=BWBQ26GVCZJFH6MIQWLJPSRYLNE46FF7ISIFUUNWKEVXQK7HWCBA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.577006872 (503h21m42.448089917s) [noise=true , address=145.79.227.40:4065 , piece_id=3K537P64SIWC7E7YUF23HFLLX63MC4QOXHSBLCSTAYMNDOAPV2OA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: <nil> Dec 26 14:50:42.577039193 (503h21m42.448122228s) [seg_pos={0 0}, upload=441726]: Segment reached optimal threshold of 14 pieces. Dec 26 14:50:42.577273911 (503h21m42.448356956s) [noise=true , address=45.91.116.13:4185 , piece_id=TDUSE3NAFPNHSXOCH7ZAD5CD4IS3YGHXK3LUPDIXX5MJRADPZDDA, seg_pos={0 0}, upload=441726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124VEt9UUm1xfNyKx1sYngqg4GnKcp6S9wtPWMairzjbQ2PpWAq): protocol: expected piece hash; context canceled; EOF Dec 26 14:50:42.577280623 (503h21m42.448363658s) [noise=true , address=45.91.116.13:4185 , piece_id=TDUSE3NAFPNHSXOCH7ZAD5CD4IS3YGHXK3LUPDIXX5MJRADPZDDA, seg_pos={0 0}, upload=441726]: Long tail cancellation due to optimal threshold Dec 26 14:50:42.577365735 (503h21m42.44844877s) [noise=true , address=77.90.31.18:4827 , piece_id=NR7KYCLNPRBMLOBJTHTXP5U7EGWHUXDJVBJPYEOX3FMFXPQC7UAQ, seg_pos={0 0}, upload=441726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 26 14:50:42.577382547 (503h21m42.448465612s) [noise=true , address=77.90.31.18:4827 , piece_id=NR7KYCLNPRBMLOBJTHTXP5U7EGWHUXDJVBJPYEOX3FMFXPQC7UAQ, seg_pos={0 0}, upload=441726]: Long tail cancellation due to optimal threshold Dec 26 14:50:42.577539466 (503h21m42.448622511s) [noise=true , address=77.90.31.14:4696 , piece_id=457RLBGIVHNB5IGVXG3N4DUENNJ5M4RU7L3TD4DJNXCS3WHUIO4Q, seg_pos={0 0}, upload=441726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK): protocol: expected piece hash; context canceled; EOF Dec 26 14:50:42.577547691 (503h21m42.448630726s) [noise=true , address=77.90.31.14:4696 , piece_id=457RLBGIVHNB5IGVXG3N4DUENNJ5M4RU7L3TD4DJNXCS3WHUIO4Q, seg_pos={0 0}, upload=441726]: Long tail cancellation due to optimal threshold Dec 26 14:50:42.577569493 (503h21m42.448652538s) [noise=true , address=77.90.31.11:4580 , piece_id=FDDSE7DP6D5FFB4DKMIKFXUYWFGHD7ZNHA3336PKTSAHVNFSVD2A, seg_pos={0 0}, upload=441726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D): protocol: expected piece hash; context canceled; EOF Dec 26 14:50:42.577578991 (503h21m42.448662036s) [noise=true , address=77.90.31.11:4580 , piece_id=FDDSE7DP6D5FFB4DKMIKFXUYWFGHD7ZNHA3336PKTSAHVNFSVD2A, seg_pos={0 0}, upload=441726]: Long tail cancellation due to optimal threshold Dec 26 14:50:42.577610290 (503h21m42.448693325s) [seg_pos={0 0}, upload=441726]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 14:50:42.577720600 (503h21m42.448803655s) [upload=441726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:50:42.577727845 (503h21m42.44881089s) [upload=441726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:50:42.880809497 (503h21m42.751892542s) [upload=441726]: Done uploading segments... Dec 26 14:50:42.880834295 (503h21m42.75191734s) [upload=441726]: Upload finished. err: <nil> Duration: 55.750285362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 11.348648447s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:53:14 UTC) orbiter warning: context deadline exceeded Duration: 11.34871364s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXXXXXXXXXXXXXXXXX..X..X] duration: 5.001779478s trace: https://jaeger.dev.storj.io/trace/47070f778b9bc951 FAIL (2025-12-26 14:53:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00180207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X...........X....XX.XXX..XXX.X....X.XX.XXX...XX..........] duration: 6.175739758s trace: https://jaeger.dev.storj.io/trace/7de0b83678515694 FAIL (2025-12-26 14:53:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ) context deadline exceeded ======LOGS====== egments... Dec 26 14:45:35.590660048 (503h16m35.461743103s) [upload=441652]: Got next segment. Inline: false Dec 26 14:45:35.590699122 (503h16m35.461782177s) [upload=441652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:45:35.590706647 (503h16m35.461789712s) [upload=441652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:45:35.842738340 (503h16m35.713821375s) [seg_pos={0 0}, upload=441652]: Begin upload segment... Dec 26 14:45:35.842786903 (503h16m35.713869938s) [seg_pos={0 0}, upload=441652]: Done begin upload segment. Dec 26 14:45:35.842815106 (503h16m35.713898151s) [noise=true , address=45.91.116.11:4120 , piece_id=JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.842894498 (503h16m35.713977533s) [noise=true , address=145.79.227.51:4409 , piece_id=7PU67WZFLRB7VKQ6B6UNYAA3GH5GPKDPFQZ2GZXL2MJLYC7QQKXA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843349324 (503h16m35.714432359s) [noise=true , address=145.79.227.49:4338 , piece_id=JHBKKSKNDQ4LG35IK3GNOLLFFPEQ3AAADPO66TJGIDZFBMKUTWOA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843718797 (503h16m35.714801832s) [noise=true , address=145.79.227.42:4121 , piece_id=HNITZLSZHAARISPYGXZPEKEKIP2FY43PJABA5TM7HMQ5M2YG5K2A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843994342 (503h16m35.715077377s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.844554508 (503h16m35.715637554s) [noise=true , address=145.79.227.14:4840 , piece_id=BTQ2VYMNB2H5JFP5AQNGKYSFMXTDKFQDT2WCOTVNHZTHAWS3GJEQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.844855001 (503h16m35.715938046s) [noise=true , address=145.79.227.5:4222 , piece_id=ZRB7VEUYC6WMDITAVXIZ6IUHEBKR5TKBP6QLY3TCLDWOXFAUVYNQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845090329 (503h16m35.716173364s) [noise=true , address=145.79.227.13:4803 , piece_id=CT3JJNSTE4ACBQYIIDFTEIDA3SHNCM2RZHVCQY62H3JWSB52IU5A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845204667 (503h16m35.716287712s) [noise=true , address=45.91.116.13:4156 , piece_id=55AO4MBH3XO5IDMOWW2U2UOR6VP3E2YTA5GBUZDFJ33X3MOQ7R2A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845572898 (503h16m35.716655933s) [noise=true , address=77.90.31.13:4629 , piece_id=6NVZXZWVYHEZ4RFXV6SX5QK4QQA3AHYI5WW5TYTU7CRIK5NH7MEQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845635296 (503h16m35.716718341s) [noise=true , address=145.79.227.6:4257 , piece_id=3667CXHPX6A7QSVZFWXWX554MAKS6XMEUIMT3G3E4YGKO6IVRMBA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846069223 (503h16m35.717152268s) [noise=true , address=145.79.227.22:4511 , piece_id=WTBEJE2XO33SIYQY2544KRMEYWGVK72FJFQJM4KY3G6XEA6MY3VA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846439247 (503h16m35.717522282s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846783553 (503h16m35.717866598s) [noise=true , address=45.91.116.6:4973 , piece_id=GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.847545484 (503h16m35.718628529s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.847695220 (503h16m35.718778255s) [noise=true , address=145.79.227.7:4294 , piece_id=HWHDWATE46DSLHWD5KU3K7GVIKKYG74G2F4OHP2FHPMGA6MXN7AQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:37.558956427 (503h16m37.430039462s) [upload=441652]: Next returned nil segment Dec 26 14:45:37.558970595 (503h16m37.43005363s) [upload=441652]: Waiting for error group managing segments... Dec 26 14:45:37.846688381 (503h16m37.717771416s) [noise=true , address=145.79.227.13:4803 , piece_id=CT3JJNSTE4ACBQYIIDFTEIDA3SHNCM2RZHVCQY62H3JWSB52IU5A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.847501870 (503h16m37.718584935s) [noise=true , address=145.79.227.5:4222 , piece_id=ZRB7VEUYC6WMDITAVXIZ6IUHEBKR5TKBP6QLY3TCLDWOXFAUVYNQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.848386425 (503h16m37.71946947s) [noise=true , address=145.79.227.51:4409 , piece_id=7PU67WZFLRB7VKQ6B6UNYAA3GH5GPKDPFQZ2GZXL2MJLYC7QQKXA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.849885419 (503h16m37.720968454s) [noise=true , address=145.79.227.42:4121 , piece_id=HNITZLSZHAARISPYGXZPEKEKIP2FY43PJABA5TM7HMQ5M2YG5K2A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.851684776 (503h16m37.722767821s) [noise=true , address=145.79.227.7:4294 , piece_id=HWHDWATE46DSLHWD5KU3K7GVIKKYG74G2F4OHP2FHPMGA6MXN7AQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.861840313 (503h16m37.732923378s) [noise=true , address=145.79.227.49:4338 , piece_id=JHBKKSKNDQ4LG35IK3GNOLLFFPEQ3AAADPO66TJGIDZFBMKUTWOA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.864286882 (503h16m37.735369917s) [noise=true , address=145.79.227.14:4840 , piece_id=BTQ2VYMNB2H5JFP5AQNGKYSFMXTDKFQDT2WCOTVNHZTHAWS3GJEQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.879425859 (503h16m37.750508905s) [noise=true , address=145.79.227.22:4511 , piece_id=WTBEJE2XO33SIYQY2544KRMEYWGVK72FJFQJM4KY3G6XEA6MY3VA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.881350424 (503h16m37.752433459s) [noise=true , address=145.79.227.6:4257 , piece_id=3667CXHPX6A7QSVZFWXWX554MAKS6XMEUIMT3G3E4YGKO6IVRMBA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.052408239 (503h16m37.923491284s) [noise=true , address=77.90.31.13:4629 , piece_id=6NVZXZWVYHEZ4RFXV6SX5QK4QQA3AHYI5WW5TYTU7CRIK5NH7MEQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.088339709 (503h16m37.959422744s) [noise=true , address=45.91.116.13:4156 , piece_id=55AO4MBH3XO5IDMOWW2U2UOR6VP3E2YTA5GBUZDFJ33X3MOQ7R2A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.092624368 (503h16m37.963707433s) [noise=true , address=45.91.116.11:4120 , piece_id=JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.203970160 (503h16m38.075053195s) [noise=true , address=45.91.116.6:4973 , piece_id=GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.203989287 (503h16m38.075072332s) [seg_pos={0 0}, upload=441652]: Segment reached optimal threshold of 13 pieces. Dec 26 14:45:38.204144872 (503h16m38.075227917s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection Dec 26 14:45:38.204151044 (503h16m38.075234099s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204230095 (503h16m38.07531314s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection Dec 26 14:45:38.204251986 (503h16m38.075335021s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204453320 (503h16m38.075536355s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 14:45:38.204461515 (503h16m38.07554456s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204479620 (503h16m38.075562655s) [seg_pos={0 0}, upload=441652]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:45:38.204555304 (503h16m38.075638339s) [upload=441652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:45:38.204560384 (503h16m38.075643419s) [upload=441652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:45:38.504709632 (503h16m38.375792677s) [upload=441652]: Done uploading segments... Dec 26 14:45:38.504726345 (503h16m38.37580938s) [upload=441652]: Upload finished. err: <nil> Duration: 6.175768573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.X...........X....XX.XXX..XXX.X....X.XX.XXX...XX.........] duration: 6.088523752s trace: https://jaeger.dev.storj.io/trace/7de0b83678515694 FAIL (2025-12-26 14:52:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ) context deadline exceeded ======LOGS====== egments... Dec 26 14:45:35.590660048 (503h16m35.461743103s) [upload=441652]: Got next segment. Inline: false Dec 26 14:45:35.590699122 (503h16m35.461782177s) [upload=441652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 14:45:35.590706647 (503h16m35.461789712s) [upload=441652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 14:45:35.842738340 (503h16m35.713821375s) [seg_pos={0 0}, upload=441652]: Begin upload segment... Dec 26 14:45:35.842786903 (503h16m35.713869938s) [seg_pos={0 0}, upload=441652]: Done begin upload segment. Dec 26 14:45:35.842815106 (503h16m35.713898151s) [noise=true , address=45.91.116.11:4120 , piece_id=JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.842894498 (503h16m35.713977533s) [noise=true , address=145.79.227.51:4409 , piece_id=7PU67WZFLRB7VKQ6B6UNYAA3GH5GPKDPFQZ2GZXL2MJLYC7QQKXA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843349324 (503h16m35.714432359s) [noise=true , address=145.79.227.49:4338 , piece_id=JHBKKSKNDQ4LG35IK3GNOLLFFPEQ3AAADPO66TJGIDZFBMKUTWOA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843718797 (503h16m35.714801832s) [noise=true , address=145.79.227.42:4121 , piece_id=HNITZLSZHAARISPYGXZPEKEKIP2FY43PJABA5TM7HMQ5M2YG5K2A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.843994342 (503h16m35.715077377s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.844554508 (503h16m35.715637554s) [noise=true , address=145.79.227.14:4840 , piece_id=BTQ2VYMNB2H5JFP5AQNGKYSFMXTDKFQDT2WCOTVNHZTHAWS3GJEQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.844855001 (503h16m35.715938046s) [noise=true , address=145.79.227.5:4222 , piece_id=ZRB7VEUYC6WMDITAVXIZ6IUHEBKR5TKBP6QLY3TCLDWOXFAUVYNQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845090329 (503h16m35.716173364s) [noise=true , address=145.79.227.13:4803 , piece_id=CT3JJNSTE4ACBQYIIDFTEIDA3SHNCM2RZHVCQY62H3JWSB52IU5A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845204667 (503h16m35.716287712s) [noise=true , address=45.91.116.13:4156 , piece_id=55AO4MBH3XO5IDMOWW2U2UOR6VP3E2YTA5GBUZDFJ33X3MOQ7R2A, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845572898 (503h16m35.716655933s) [noise=true , address=77.90.31.13:4629 , piece_id=6NVZXZWVYHEZ4RFXV6SX5QK4QQA3AHYI5WW5TYTU7CRIK5NH7MEQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.845635296 (503h16m35.716718341s) [noise=true , address=145.79.227.6:4257 , piece_id=3667CXHPX6A7QSVZFWXWX554MAKS6XMEUIMT3G3E4YGKO6IVRMBA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846069223 (503h16m35.717152268s) [noise=true , address=145.79.227.22:4511 , piece_id=WTBEJE2XO33SIYQY2544KRMEYWGVK72FJFQJM4KY3G6XEA6MY3VA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846439247 (503h16m35.717522282s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.846783553 (503h16m35.717866598s) [noise=true , address=45.91.116.6:4973 , piece_id=GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.847545484 (503h16m35.718628529s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:35.847695220 (503h16m35.718778255s) [noise=true , address=145.79.227.7:4294 , piece_id=HWHDWATE46DSLHWD5KU3K7GVIKKYG74G2F4OHP2FHPMGA6MXN7AQ, seg_pos={0 0}, upload=441652]: Uploading piece... Dec 26 14:45:37.558956427 (503h16m37.430039462s) [upload=441652]: Next returned nil segment Dec 26 14:45:37.558970595 (503h16m37.43005363s) [upload=441652]: Waiting for error group managing segments... Dec 26 14:45:37.846688381 (503h16m37.717771416s) [noise=true , address=145.79.227.13:4803 , piece_id=CT3JJNSTE4ACBQYIIDFTEIDA3SHNCM2RZHVCQY62H3JWSB52IU5A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.847501870 (503h16m37.718584935s) [noise=true , address=145.79.227.5:4222 , piece_id=ZRB7VEUYC6WMDITAVXIZ6IUHEBKR5TKBP6QLY3TCLDWOXFAUVYNQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.848386425 (503h16m37.71946947s) [noise=true , address=145.79.227.51:4409 , piece_id=7PU67WZFLRB7VKQ6B6UNYAA3GH5GPKDPFQZ2GZXL2MJLYC7QQKXA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.849885419 (503h16m37.720968454s) [noise=true , address=145.79.227.42:4121 , piece_id=HNITZLSZHAARISPYGXZPEKEKIP2FY43PJABA5TM7HMQ5M2YG5K2A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.851684776 (503h16m37.722767821s) [noise=true , address=145.79.227.7:4294 , piece_id=HWHDWATE46DSLHWD5KU3K7GVIKKYG74G2F4OHP2FHPMGA6MXN7AQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.861840313 (503h16m37.732923378s) [noise=true , address=145.79.227.49:4338 , piece_id=JHBKKSKNDQ4LG35IK3GNOLLFFPEQ3AAADPO66TJGIDZFBMKUTWOA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.864286882 (503h16m37.735369917s) [noise=true , address=145.79.227.14:4840 , piece_id=BTQ2VYMNB2H5JFP5AQNGKYSFMXTDKFQDT2WCOTVNHZTHAWS3GJEQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.879425859 (503h16m37.750508905s) [noise=true , address=145.79.227.22:4511 , piece_id=WTBEJE2XO33SIYQY2544KRMEYWGVK72FJFQJM4KY3G6XEA6MY3VA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:37.881350424 (503h16m37.752433459s) [noise=true , address=145.79.227.6:4257 , piece_id=3667CXHPX6A7QSVZFWXWX554MAKS6XMEUIMT3G3E4YGKO6IVRMBA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.052408239 (503h16m37.923491284s) [noise=true , address=77.90.31.13:4629 , piece_id=6NVZXZWVYHEZ4RFXV6SX5QK4QQA3AHYI5WW5TYTU7CRIK5NH7MEQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.088339709 (503h16m37.959422744s) [noise=true , address=45.91.116.13:4156 , piece_id=55AO4MBH3XO5IDMOWW2U2UOR6VP3E2YTA5GBUZDFJ33X3MOQ7R2A, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.092624368 (503h16m37.963707433s) [noise=true , address=45.91.116.11:4120 , piece_id=JDVLYT3XOAK5NBPZFEJLYLCCAUOX2XGAXCHGIPUN5MVW2H6M2TRQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.203970160 (503h16m38.075053195s) [noise=true , address=45.91.116.6:4973 , piece_id=GTHV5DFKCF2TJSWKRNPAMLHT5NRPZMOSI7IBQACLDKICXCP3MXHQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: <nil> Dec 26 14:45:38.203989287 (503h16m38.075072332s) [seg_pos={0 0}, upload=441652]: Segment reached optimal threshold of 13 pieces. Dec 26 14:45:38.204144872 (503h16m38.075227917s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45146->45.91.116.14:4189: use of closed network connection Dec 26 14:45:38.204151044 (503h16m38.075234099s) [noise=true , address=45.91.116.14:4189 , piece_id=ZLLP2J7XWYU7SYKDIJSEU7OTFDZI7BPEQV7K5JN4R3SBXE64JPQQ, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204230095 (503h16m38.07531314s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53576->45.91.116.5:4300: use of closed network connection Dec 26 14:45:38.204251986 (503h16m38.075335021s) [noise=true , address=45.91.116.5:4300 , piece_id=GJ57XDAWHHXJTIL6B3XBUEOI4UFAVDVOK6GLSQ5EVMLXOECO4OLA, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204453320 (503h16m38.075536355s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 14:45:38.204461515 (503h16m38.07554456s) [noise=true , address=45.91.116.7:4009 , piece_id=INVXC25YZU3VZGNMHXWVQOWJFRVVDGQTLIFLCGVIJ2XC7KNCKSAA, seg_pos={0 0}, upload=441652]: Long tail cancellation due to optimal threshold Dec 26 14:45:38.204479620 (503h16m38.075562655s) [seg_pos={0 0}, upload=441652]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 14:45:38.204555304 (503h16m38.075638339s) [upload=441652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 14:45:38.204560384 (503h16m38.075643419s) [upload=441652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 14:45:38.504709632 (503h16m38.375792677s) [upload=441652]: Done uploading segments... Dec 26 14:45:38.504726345 (503h16m38.37580938s) [upload=441652]: Upload finished. err: <nil> Duration: 6.088550793s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 14.125033252s trace: https://jaeger.dev.storj.io/trace/3ccdb98964ad100c FAIL (2025-12-26 14:52:33 UTC) orbiter warning: context deadline exceeded Duration: 14.125054723s edge gateway us-select-1

1
...
11395
11396
11397
11398
11399
...
17908