Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX..........] duration:
5.77323115s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:50:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.773265886s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.096296272s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:50:02 UTC)
orbiter warning: context deadline exceeded
Duration: 6.096325579s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
41.388880809s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:49:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 41.388900467s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
3.685320074s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:49:51 UTC)
orbiter warning: context deadline exceeded
Duration: 3.685367413s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.916594127s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:49:47 UTC)
orbiter warning: context deadline exceeded
Duration: 5.916619385s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
3.466638575s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:49:35 UTC)
orbiter warning: context deadline exceeded
Duration: 3.466663032s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX.........] duration:
5.759995187s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:49:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.760013923s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
31.688189698s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:49:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 31.688208595s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [..X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX........] duration:
6.007045613s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:48:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 6.007068666s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
47.172908629s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:48:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 47.17295125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.945844896s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:48:38 UTC)
orbiter warning: context deadline exceeded
Duration: 5.945869753s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
3.43122412s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:48:25 UTC)
orbiter warning: context deadline exceeded
Duration: 3.431253505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [...X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX.......] duration:
5.923175712s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:48:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.923204416s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.914716237s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:47:52 UTC)
orbiter warning: context deadline exceeded
Duration: 6.914745013s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
28.789867207s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:47:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 28.789896602s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [....X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX......] duration:
6.083568346s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:47:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 6.083617299s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
6.374066566s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:47:27 UTC)
orbiter warning: context deadline exceeded
Duration: 6.374085432s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
33.515368337s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:47:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 33.515404345s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
3.485055565s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:47:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.485112874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
24.551190216s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:47:03 UTC)
orbiter warning: context deadline exceeded
Duration: 24.551225031s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.....X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX.....] duration:
6.00516272s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:46:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 6.005189811s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.913691301s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:46:13 UTC)
orbiter warning: context deadline exceeded
Duration: 5.913745214s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
3.436996221s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:46:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.437019425s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
3.472986594s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:45:56 UTC)
orbiter warning: context deadline exceeded
Duration: 3.473013094s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
23.484945709s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:45:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 23.484993821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [......X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX....] duration:
5.739274301s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:45:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.739302715s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
3.476187398s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:45:39 UTC)
orbiter warning: context deadline exceeded
Duration: 3.476211865s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.895213406s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:45:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.895241609s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
36.236286869s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:45:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 36.236303894s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.......X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX...] duration:
5.877429742s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:45:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.877462334s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
3.549029122s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:45:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.549054741s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
6.13609161s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:44:48 UTC)
orbiter warning: context deadline exceeded
Duration: 6.136114815s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
47.617194898s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:44:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 47.617221407s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
3.481320031s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:44:19 UTC)
orbiter warning: context deadline exceeded
Duration: 3.481346932s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [........X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX..] duration:
5.783650131s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:44:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.783693823s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
3.476293711s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:44:02 UTC)
orbiter warning: context deadline exceeded
Duration: 3.476322325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
45.182073902s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:43:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 45.182105943s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.074982296s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:43:46 UTC)
orbiter warning: context deadline exceeded
Duration: 6.075006333s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
3.518585624s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:43:20 UTC)
orbiter warning: context deadline exceeded
Duration: 3.518617002s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.........X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX.] duration:
5.943096495s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:43:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 5.943126391s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.221161387s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:42:59 UTC)
orbiter warning: context deadline exceeded
Duration: 6.221188049s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
36.350598549s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:42:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 36.350628556s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [..........X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XXX] duration:
30.003223687s trace:
https://jaeger.dev.storj.io/trace/3312cc3668ef2c5f
FAIL (2025-12-26 08:42:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 08:42:18.506798665 (497h13m18.37788171s) [upload=436400]: Starting upload
Dec 26 08:42:18.506881102 (497h13m18.377964137s) [upload=436400]: Done starting upload
Dec 26 08:42:18.506963669 (497h13m18.378046714s) [upload=436400]: Uploading segments...
Dec 26 08:42:18.507170413 (497h13m18.378253478s) [upload=436400]: Got next segment. Inline: false
Dec 26 08:42:18.507247920 (497h13m18.378330965s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:42:18.507255454 (497h13m18.378338499s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:42:18.754283459 (497h13m18.625366504s) [seg_pos={0 0}, upload=436400]: Begin upload segment...
Dec 26 08:42:18.754356328 (497h13m18.625439373s) [seg_pos={0 0}, upload=436400]: Done begin upload segment.
Dec 26 08:42:18.754391745 (497h13m18.62547479s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.754466177 (497h13m18.625549222s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755567554 (497h13m18.626650599s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.755910858 (497h13m18.626993893s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756229915 (497h13m18.62731296s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.756613886 (497h13m18.627696922s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757020902 (497h13m18.628103937s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757428849 (497h13m18.628511904s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757902380 (497h13m18.628985425s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.757979276 (497h13m18.629062321s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758365802 (497h13m18.629448837s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.758773228 (497h13m18.629856263s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.759342462 (497h13m18.630425507s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760026635 (497h13m18.63110968s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.760798124 (497h13m18.631881169s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:18.761677319 (497h13m18.632760364s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Uploading piece...
Dec 26 08:42:20.466851608 (497h13m20.337934643s) [upload=436400]: Next returned nil segment
Dec 26 08:42:20.466865273 (497h13m20.337948318s) [upload=436400]: Waiting for error group managing segments...
Dec 26 08:42:20.750402218 (497h13m20.621485253s) [noise=true , address=145.79.227.37:4567 , piece_id=J7Z3543ZQ2TIGFFHBO7V7GJKTGOY54Z6WAPYDI7ALYRYDS6NEK2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.763579740 (497h13m20.634662785s) [noise=true , address=145.79.227.48:4301 , piece_id=IS2W2WAK7NOMKFEHY3QK7KS35XPQYC2723Q2ZJ27HTX4FPBQYHDQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.767844120 (497h13m20.638927155s) [noise=true , address=145.79.227.38:4977 , piece_id=MYMT3XBOXWU5DZOTPVHSVVXOKNHW7WWVCBECIO37VAYUTFK7XZSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.771941653 (497h13m20.643024698s) [noise=true , address=145.79.227.16:4912 , piece_id=TIPMXPQDX4HVH2NUQ3FUFT6J34SNM5DT2NHDXCZO76MMGPEEEHMQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.773887688 (497h13m20.644970723s) [noise=true , address=145.79.227.17:4970 , piece_id=JXWYFTTKYNB6BSPBKWBK4YPF2PNAGWJKXHJAOUFIJVVIKIVNPUJA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781414641 (497h13m20.652497676s) [noise=true , address=145.79.227.51:4411 , piece_id=7JDMY4F74SKL6NSFN2HO36EGUCM5MWVWR5HLXACHBLZXTDRRWI3Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.781591407 (497h13m20.652674452s) [noise=true , address=145.79.227.46:4603 , piece_id=7L43455SH4JVSXBJYL3WVMRCWFQZ2CEI6DQA7MUXDYSKPPKESY7A, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:20.787415879 (497h13m20.658498924s) [noise=true , address=145.79.227.45:4229 , piece_id=GNXPJKQLV6ITICMBLE4PY2ODOKEJ6WWBUAPA74AO4IBVOVVEFSLA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.029225186 (497h13m20.900308221s) [noise=true , address=45.91.116.4:4937 , piece_id=7HT3WA63RZNVO3AX775OL37N4BJRXZCZ5P6A2A2Z6NPZX6RASSSQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.124342636 (497h13m20.995425671s) [noise=true , address=77.90.31.21:4878 , piece_id=RNZ3EK2ZERZY6IQEBAZPHEUDBHNF7BARJF5FVZMLVVMLH5OK7WCQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.148939092 (497h13m21.020022127s) [noise=true , address=77.90.31.14:4664 , piece_id=JFEBDNYO2OQRCKCLBKWS43SZCZ7L2LSIF5FP2JS37FFARUEHCESA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.733228030 (497h13m21.604311075s) [noise=true , address=77.90.31.11:4555 , piece_id=ZTSV63NMIHDFU464BAUD3MH3645MJOCGLTDQEKIRPD47EMLLO5EA, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818455067 (497h13m21.689538102s) [noise=true , address=77.90.31.17:4773 , piece_id=P645R4U4WSCWJR5XZSGFGVVF3QHAVSP6SN46CPQGBT6JWJB57ZIQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: <nil>
Dec 26 08:42:21.818474194 (497h13m21.689557239s) [seg_pos={0 0}, upload=436400]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:42:21.818682150 (497h13m21.689765195s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818690926 (497h13m21.689773971s) [noise=true , address=77.90.31.10:4517 , piece_id=RD3ARVUTAOQRS3YUQ7WL2LF47CXQFSVDDJSJUAEZAFVGUU4HJ7KQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818911596 (497h13m21.689994631s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818918088 (497h13m21.690001123s) [noise=true , address=145.79.227.40:4051 , piece_id=NLPGNO63SJLYA65HYN2CJBJFS6OTLX7ISULHSM6Q54PWOOTVAVWQ, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.818941363 (497h13m21.690024418s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 26 08:42:21.818997390 (497h13m21.690080425s) [noise=true , address=77.90.31.18:4808 , piece_id=INW543LSVQLUQQ4FZZXVTVPV2LSGPKMUVHCSKE3OQ2CQAIEQFD2Q, seg_pos={0 0}, upload=436400]: Long tail cancellation due to optimal threshold
Dec 26 08:42:21.819032707 (497h13m21.690115742s) [seg_pos={0 0}, upload=436400]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:42:21.819224622 (497h13m21.690307657s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:42:21.819232357 (497h13m21.690315392s) [upload=436400]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:42:22.109308312 (497h13m21.980391357s) [upload=436400]: Done uploading segments...
Dec 26 08:42:22.109331417 (497h13m21.980414452s) [upload=436400]: Upload finished. err: <nil>
Duration: 30.003253295s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
3.590877076s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:42:43 UTC)
orbiter warning: context deadline exceeded
Duration: 3.590903167s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [...........X..X.XX.X.X..X..X.XXX.....X......X.X...X.XX..X.XX] duration:
30.002500293s trace:
https://jaeger.dev.storj.io/trace/3a75ed4065de85a8
FAIL (2025-12-26 08:42:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3U4VEU3U5LGNREBFQDS27AX66FAADSK7RYCI2FSQ4SCOMYYLNN4A) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: K3GGNL2ATRKFNVGJ63OXRCHHXZG2JO3JJU6RVDZ5M5D2Q3NN2LJQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: JYMIYZGEYORKZL7RXT7QINX5RXVBBIH735Q7JW22AKZFIOFDL7UA) context deadline exceeded
======LOGS======
Dec 26 08:41:39.754811332 (497h12m39.625894377s) [upload=436385]: Starting upload
Dec 26 08:41:39.754890433 (497h12m39.625973478s) [upload=436385]: Done starting upload
Dec 26 08:41:39.755100653 (497h12m39.626183698s) [upload=436385]: Uploading segments...
Dec 26 08:41:39.755164044 (497h12m39.626247099s) [upload=436385]: Got next segment. Inline: false
Dec 26 08:41:39.755202296 (497h12m39.626285331s) [upload=436385]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 08:41:39.755208348 (497h12m39.626291383s) [upload=436385]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 08:41:39.999183890 (497h12m39.870266935s) [seg_pos={0 0}, upload=436385]: Begin upload segment...
Dec 26 08:41:39.999243493 (497h12m39.870326529s) [seg_pos={0 0}, upload=436385]: Done begin upload segment.
Dec 26 08:41:39.999281927 (497h12m39.870364982s) [noise=true , address=145.79.227.5:4223 , piece_id=2KC4WMA745RUDV6NTP2NH74TOOXIXCCV72ICWK6H57JT7LI3IWSQ, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:39.999296704 (497h12m39.870379739s) [noise=true , address=145.79.227.19:4079 , piece_id=KVDN5AY5FJ4DKKAOSAMPTAKXA3BW57GV2DMXDVJ65KECUZJS2Z3Q, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:39.999957342 (497h12m39.871040377s) [noise=true , address=77.90.31.15:4697 , piece_id=U3JANLEUA77LV5NNEXMFE5UTDYQEZZRVNUL6ELNMDJMQ7B7XS5KA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.000106918 (497h12m39.871189953s) [noise=true , address=145.79.227.32:4834 , piece_id=OCZGVYEKHMJXE5CO663MPJLF7ZWUQUTZT24TMOC5YVXRFQQ2JJCA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.000513191 (497h12m39.871596236s) [noise=true , address=145.79.227.10:4403 , piece_id=TMEUYHOB2Y6LXOQZKDK6JA7IHZSI5MG63G4CHNRHE25G4VXLWG3A, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.000897593 (497h12m39.871980638s) [noise=true , address=45.91.116.14:4192 , piece_id=R75D4UOQPUDFYDSIWKQZNDKGJAEGT2RKVHWLDONYQSBGS7FKNLVA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.001258200 (497h12m39.872341245s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3QXNY3JN2QLSFJUUHE3C5UIOFYGFS2GTSP635IMK6USH7TVTUXA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.001607315 (497h12m39.87269035s) [noise=true , address=145.79.227.33:4869 , piece_id=GLMBU5HVI7FDDGD2M3L5ML56TGNSENZ3DIWTKCOAFU5GCF4TNFRA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.002034117 (497h12m39.873117162s) [noise=true , address=45.91.116.6:4975 , piece_id=3U4VEU3U5LGNREBFQDS27AX66FAADSK7RYCI2FSQ4SCOMYYLNN4A, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.002491568 (497h12m39.873574603s) [noise=true , address=145.79.227.38:4979 , piece_id=RSTUD5XUH7BR4MLNZEF6RCTEQHX4JR45EOROZOENAZDZDJN63U4A, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.002921537 (497h12m39.874004573s) [noise=true , address=45.91.116.11:4119 , piece_id=EB3YNXKLDOHSYBTME3RXS5DO6SVIXRJNVZD2NSC5RIL35K6WUBMQ, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.003321178 (497h12m39.874404233s) [noise=true , address=77.90.31.11:4553 , piece_id=PRTIXAKHZLRZDLVOTSQEBXGAR2KKE52EJB7VKMRSMVXB5ZJYMMKA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.003766276 (497h12m39.874849311s) [noise=true , address=45.91.116.12:4225 , piece_id=K3GGNL2ATRKFNVGJ63OXRCHHXZG2JO3JJU6RVDZ5M5D2Q3NN2LJQ, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.004202786 (497h12m39.875285831s) [noise=true , address=145.79.227.43:4158 , piece_id=AT24CKYX343R63HDR3FYKEJG7A6IHVWXE3B46WD3R6NZ6MCAHSHA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.004681298 (497h12m39.875764343s) [noise=true , address=77.90.31.10:4519 , piece_id=JYMIYZGEYORKZL7RXT7QINX5RXVBBIH735Q7JW22AKZFIOFDL7UA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:40.005185338 (497h12m39.876268373s) [noise=true , address=145.79.227.13:4802 , piece_id=EOQH6Z3DNHYSJMVHHBWVHFBSF4JQKEEQVVEPPF6TYXBDV7OF7BFA, seg_pos={0 0}, upload=436385]: Uploading piece...
Dec 26 08:41:41.225124735 (497h12m41.09620777s) [noise=true , address=77.90.31.15:4697 , piece_id=U3JANLEUA77LV5NNEXMFE5UTDYQEZZRVNUL6ELNMDJMQ7B7XS5KA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000
Dec 26 08:41:41.722443478 (497h12m41.593526513s) [upload=436385]: Next returned nil segment
Dec 26 08:41:41.722454699 (497h12m41.593537734s) [upload=436385]: Waiting for error group managing segments...
Dec 26 08:41:42.003644463 (497h12m41.874727498s) [noise=true , address=145.79.227.13:4802 , piece_id=EOQH6Z3DNHYSJMVHHBWVHFBSF4JQKEEQVVEPPF6TYXBDV7OF7BFA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.018308696 (497h12m41.889391741s) [noise=true , address=145.79.227.7:4293 , piece_id=Q3QXNY3JN2QLSFJUUHE3C5UIOFYGFS2GTSP635IMK6USH7TVTUXA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.019347815 (497h12m41.89043086s) [noise=true , address=145.79.227.38:4979 , piece_id=RSTUD5XUH7BR4MLNZEF6RCTEQHX4JR45EOROZOENAZDZDJN63U4A, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.027768640 (497h12m41.898851675s) [noise=true , address=145.79.227.5:4223 , piece_id=2KC4WMA745RUDV6NTP2NH74TOOXIXCCV72ICWK6H57JT7LI3IWSQ, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.034410327 (497h12m41.905493362s) [noise=true , address=145.79.227.19:4079 , piece_id=KVDN5AY5FJ4DKKAOSAMPTAKXA3BW57GV2DMXDVJ65KECUZJS2Z3Q, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.034492924 (497h12m41.905575959s) [noise=true , address=145.79.227.10:4403 , piece_id=TMEUYHOB2Y6LXOQZKDK6JA7IHZSI5MG63G4CHNRHE25G4VXLWG3A, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.042390482 (497h12m41.913473527s) [noise=true , address=145.79.227.32:4834 , piece_id=OCZGVYEKHMJXE5CO663MPJLF7ZWUQUTZT24TMOC5YVXRFQQ2JJCA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.044373167 (497h12m41.915456213s) [noise=true , address=145.79.227.43:4158 , piece_id=AT24CKYX343R63HDR3FYKEJG7A6IHVWXE3B46WD3R6NZ6MCAHSHA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.361607968 (497h12m42.232691013s) [noise=true , address=77.90.31.10:4519 , piece_id=JYMIYZGEYORKZL7RXT7QINX5RXVBBIH735Q7JW22AKZFIOFDL7UA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.371305083 (497h12m42.242388118s) [noise=true , address=45.91.116.6:4975 , piece_id=3U4VEU3U5LGNREBFQDS27AX66FAADSK7RYCI2FSQ4SCOMYYLNN4A, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.452064644 (497h12m42.323147679s) [noise=true , address=45.91.116.12:4225 , piece_id=K3GGNL2ATRKFNVGJ63OXRCHHXZG2JO3JJU6RVDZ5M5D2Q3NN2LJQ, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:42.599995399 (497h12m42.471078434s) [noise=true , address=45.91.116.14:4192 , piece_id=R75D4UOQPUDFYDSIWKQZNDKGJAEGT2RKVHWLDONYQSBGS7FKNLVA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:43.151072065 (497h12m43.0221551s) [noise=true , address=45.91.116.11:4119 , piece_id=EB3YNXKLDOHSYBTME3RXS5DO6SVIXRJNVZD2NSC5RIL35K6WUBMQ, seg_pos={0 0}, upload=436385]: Done uploading piece. err: <nil>
Dec 26 08:41:43.151095160 (497h12m43.022178205s) [seg_pos={0 0}, upload=436385]: Segment reached optimal threshold of 13 pieces.
Dec 26 08:41:43.151328473 (497h12m43.022411508s) [noise=true , address=145.79.227.33:4869 , piece_id=GLMBU5HVI7FDDGD2M3L5ML56TGNSENZ3DIWTKCOAFU5GCF4TNFRA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 26 08:41:43.151338433 (497h12m43.022421478s) [noise=true , address=145.79.227.33:4869 , piece_id=GLMBU5HVI7FDDGD2M3L5ML56TGNSENZ3DIWTKCOAFU5GCF4TNFRA, seg_pos={0 0}, upload=436385]: Long tail cancellation due to optimal threshold
Dec 26 08:41:43.151426500 (497h12m43.022509545s) [noise=true , address=77.90.31.11:4553 , piece_id=PRTIXAKHZLRZDLVOTSQEBXGAR2KKE52EJB7VKMRSMVXB5ZJYMMKA, seg_pos={0 0}, upload=436385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:53494->77.90.31.11:4553: use of closed network connection; storage node overloaded, request limit: 2000
Dec 26 08:41:43.151447791 (497h12m43.022530846s) [noise=true , address=77.90.31.11:4553 , piece_id=PRTIXAKHZLRZDLVOTSQEBXGAR2KKE52EJB7VKMRSMVXB5ZJYMMKA, seg_pos={0 0}, upload=436385]: Long tail cancellation due to optimal threshold
Dec 26 08:41:43.151547070 (497h12m43.022630115s) [seg_pos={0 0}, upload=436385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 08:41:43.151650798 (497h12m43.022733843s) [upload=436385]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:41:43.151656569 (497h12m43.022739604s) [upload=436385]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:41:43.440039961 (497h12m43.311123006s) [upload=436385]: Done uploading segments...
Dec 26 08:41:43.440077573 (497h12m43.311160608s) [upload=436385]: Upload finished. err: <nil>
Duration: 30.002526343s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
3.453528863s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:41:56 UTC)
orbiter warning: context deadline exceeded
Duration: 3.453589508s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
6.244095448s trace:
https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4
FAIL (2025-12-26 08:41:51 UTC)
orbiter warning: context deadline exceeded
Duration: 6.244121628s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
32.776184571s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:41:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 32.776197916s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
52.026410745s trace:
https://jaeger.dev.storj.io/trace/6820217d73ecf2a
FAIL (2025-12-26 08:41:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece...
Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment
Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments...
Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil>
Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces.
Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF
Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold
Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments...
Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil>
Duration: 52.026492672s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
3.405256103s trace:
https://jaeger.dev.storj.io/trace/185fddc4dc40eba9
FAIL (2025-12-26 08:40:59 UTC)
orbiter warning: context deadline exceeded
Duration: 3.405283584s
core
uplink
us-select-1
us1-placement-20
|