Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......] duration: 3.517932364s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:57:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517953964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.147930184s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:57:34 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.147958267s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXX] duration: 30.002161041s trace: https://jaeger.dev.storj.io/trace/534c43dc58fae9e0 FAIL (2025-12-28 04:57:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 28 04:56:40.981021111 (541h27m40.852104156s) [upload=474834]: Got next segment. Inline: false Dec 28 04:56:40.981049044 (541h27m40.852132079s) [upload=474834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:56:40.981053092 (541h27m40.852136127s) [upload=474834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:56:41.227027684 (541h27m41.098110739s) [seg_pos={0 0}, upload=474834]: Begin upload segment... Dec 28 04:56:41.227084923 (541h27m41.098167968s) [seg_pos={0 0}, upload=474834]: Done begin upload segment. Dec 28 04:56:41.227122324 (541h27m41.098205359s) [noise=true , address=145.79.227.27:4691 , piece_id=WAJCFDNSLUBE54MGTNMCPSKATWEVWFNNYWI3R5LHMIXPX7NOSOYQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.227129869 (541h27m41.098212924s) [noise=true , address=45.91.116.11:4119 , piece_id=HU32YN4OSCWMUJTH2VTZQE2QXNL3QRMYAL6L52KEMIW5MV2G74FQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.227629009 (541h27m41.098712054s) [noise=true , address=145.79.227.37:4566 , piece_id=RABW3MU64YTVCQB5BZY6XUUFF5R6L27G7WE6AKRYZ5ZTFLUIXJ7A, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.228053758 (541h27m41.099136803s) [noise=true , address=45.91.116.12:4227 , piece_id=GSXYGAIHYXIMF7BWAUBZIBFTUX5DB5LQB2GJUIGW75XE33DM3WXQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.228480770 (541h27m41.099563816s) [noise=true , address=145.79.227.5:4222 , piece_id=RFLIP6Z5VMN4S3HMFOMAGEE5KBQNFIX6B4B3PQGBGFCABWIOPR6Q, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.229405762 (541h27m41.100488807s) [noise=true , address=45.91.116.10:4081 , piece_id=QA5A73OZBVQFS732MIBCBO6TE2NU4RY6UEZJRQU2C7OJ7MGZ2CAQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.229785485 (541h27m41.10086852s) [noise=true , address=45.91.116.9:4261 , piece_id=5ECT436ZVUAQQOVCRBGSYTHC6SNENOKP4N7C3PNVVGCMVZFMX2HQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.230182631 (541h27m41.101265676s) [noise=true , address=45.91.116.8:4045 , piece_id=M2WLLT7SAUK3UEG4Z3BB7NODPKLZL3QIMB4QLPTN5KTF4F4BCWCA, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.231349713 (541h27m41.102432749s) [noise=true , address=45.91.116.13:4154 , piece_id=2SF7DISCMAISR4VMDUZ24OSWZ3JA5F5NG5CQ3CEMY4R2G6GNOA3A, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.232537556 (541h27m41.103620591s) [noise=true , address=145.79.227.14:4840 , piece_id=DPCWONW6LKAK4OKMG5RN3UV3YAG3RVP6WSJJDJYLYEG43265K24Q, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.233671354 (541h27m41.104754399s) [noise=true , address=77.90.31.15:4697 , piece_id=U5XDAZJJFY57KX267WDN3HYSIH6DOGK5DD3JJLBWUFSQUWWTHVBA, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.235109243 (541h27m41.106192278s) [noise=true , address=145.79.227.17:4971 , piece_id=CYZDA6CI5XMHMMXVMCVKOW3AXPZXPWGLHQ2L5WX35PIWM3D4JRUQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.236492216 (541h27m41.107575251s) [noise=true , address=145.79.227.50:4374 , piece_id=42QSBJTZMMVLYOB75ACLP6KKLX3NTB6N5TADCNRBB2SMPYYQOVOA, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.237888194 (541h27m41.108971229s) [noise=true , address=77.90.31.14:4663 , piece_id=SSDCZJQB4CISX4NMLOQAEOS2OHCKKW2V2Q33Z3HWTXPFOEHYHLTQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.239095343 (541h27m41.110178388s) [noise=true , address=145.79.227.2:4114 , piece_id=SQCA3M6GXUVTNFA7OMBBLIXTLQVQ2FKMP2SA7Y2UHC5VRQWY7I4Q, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.239653235 (541h27m41.11073628s) [noise=true , address=77.90.31.21:4878 , piece_id=3RQ2QIKITNFDWEJYHMJMFACJSYQA57ZLMWDH374GSVMVCDPOYBSA, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:42.933914392 (541h27m42.804997437s) [upload=474834]: Next returned nil segment Dec 28 04:56:42.933934900 (541h27m42.805017936s) [upload=474834]: Waiting for error group managing segments... Dec 28 04:56:43.224234625 (541h27m43.09531767s) [noise=true , address=145.79.227.27:4691 , piece_id=WAJCFDNSLUBE54MGTNMCPSKATWEVWFNNYWI3R5LHMIXPX7NOSOYQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.229268190 (541h27m43.100351225s) [noise=true , address=145.79.227.5:4222 , piece_id=RFLIP6Z5VMN4S3HMFOMAGEE5KBQNFIX6B4B3PQGBGFCABWIOPR6Q, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.230062331 (541h27m43.101145377s) [noise=true , address=145.79.227.37:4566 , piece_id=RABW3MU64YTVCQB5BZY6XUUFF5R6L27G7WE6AKRYZ5ZTFLUIXJ7A, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.241520139 (541h27m43.112603184s) [noise=true , address=145.79.227.2:4114 , piece_id=SQCA3M6GXUVTNFA7OMBBLIXTLQVQ2FKMP2SA7Y2UHC5VRQWY7I4Q, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.245773718 (541h27m43.116856753s) [noise=true , address=145.79.227.14:4840 , piece_id=DPCWONW6LKAK4OKMG5RN3UV3YAG3RVP6WSJJDJYLYEG43265K24Q, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.249772373 (541h27m43.120855408s) [noise=true , address=145.79.227.17:4971 , piece_id=CYZDA6CI5XMHMMXVMCVKOW3AXPZXPWGLHQ2L5WX35PIWM3D4JRUQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.263589753 (541h27m43.134672798s) [noise=true , address=145.79.227.50:4374 , piece_id=42QSBJTZMMVLYOB75ACLP6KKLX3NTB6N5TADCNRBB2SMPYYQOVOA, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.410124066 (541h27m43.281207102s) [noise=true , address=77.90.31.21:4878 , piece_id=3RQ2QIKITNFDWEJYHMJMFACJSYQA57ZLMWDH374GSVMVCDPOYBSA, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.431508726 (541h27m43.302591761s) [noise=true , address=77.90.31.15:4697 , piece_id=U5XDAZJJFY57KX267WDN3HYSIH6DOGK5DD3JJLBWUFSQUWWTHVBA, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.523009627 (541h27m43.394092672s) [noise=true , address=77.90.31.14:4663 , piece_id=SSDCZJQB4CISX4NMLOQAEOS2OHCKKW2V2Q33Z3HWTXPFOEHYHLTQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.588758443 (541h27m43.459841488s) [noise=true , address=45.91.116.10:4081 , piece_id=QA5A73OZBVQFS732MIBCBO6TE2NU4RY6UEZJRQU2C7OJ7MGZ2CAQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.602212371 (541h27m43.473295406s) [noise=true , address=45.91.116.9:4261 , piece_id=5ECT436ZVUAQQOVCRBGSYTHC6SNENOKP4N7C3PNVVGCMVZFMX2HQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.633373786 (541h27m43.504456821s) [noise=true , address=45.91.116.12:4227 , piece_id=GSXYGAIHYXIMF7BWAUBZIBFTUX5DB5LQB2GJUIGW75XE33DM3WXQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.633391681 (541h27m43.504474716s) [seg_pos={0 0}, upload=474834]: Segment reached optimal threshold of 13 pieces. Dec 28 04:56:43.633588164 (541h27m43.504671199s) [noise=true , address=45.91.116.11:4119 , piece_id=HU32YN4OSCWMUJTH2VTZQE2QXNL3QRMYAL6L52KEMIW5MV2G74FQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:39374->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:39374->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39374->45.91.116.11:4119: use of closed network connection Dec 28 04:56:43.633595048 (541h27m43.504678083s) [noise=true , address=45.91.116.11:4119 , piece_id=HU32YN4OSCWMUJTH2VTZQE2QXNL3QRMYAL6L52KEMIW5MV2G74FQ, seg_pos={0 0}, upload=474834]: Long tail cancellation due to optimal threshold Dec 28 04:56:43.633859120 (541h27m43.504942156s) [noise=true , address=45.91.116.13:4154 , piece_id=2SF7DISCMAISR4VMDUZ24OSWZ3JA5F5NG5CQ3CEMY4R2G6GNOA3A, seg_pos={0 0}, upload=474834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 04:56:43.633865643 (541h27m43.504948678s) [noise=true , address=45.91.116.13:4154 , piece_id=2SF7DISCMAISR4VMDUZ24OSWZ3JA5F5NG5CQ3CEMY4R2G6GNOA3A, seg_pos={0 0}, upload=474834]: Long tail cancellation due to optimal threshold Dec 28 04:56:43.633879099 (541h27m43.504962134s) [noise=true , address=45.91.116.8:4045 , piece_id=M2WLLT7SAUK3UEG4Z3BB7NODPKLZL3QIMB4QLPTN5KTF4F4BCWCA, seg_pos={0 0}, upload=474834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:39190->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:39190->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39190->45.91.116.8:4045: use of closed network connection Dec 28 04:56:43.633894919 (541h27m43.504977954s) [noise=true , address=45.91.116.8:4045 , piece_id=M2WLLT7SAUK3UEG4Z3BB7NODPKLZL3QIMB4QLPTN5KTF4F4BCWCA, seg_pos={0 0}, upload=474834]: Long tail cancellation due to optimal threshold Dec 28 04:56:43.633917813 (541h27m43.505000858s) [seg_pos={0 0}, upload=474834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:56:43.634018354 (541h27m43.505101389s) [upload=474834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:56:43.634026109 (541h27m43.505109164s) [upload=474834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:56:43.931901748 (541h27m43.802984783s) [upload=474834]: Done uploading segments... Dec 28 04:56:43.931919301 (541h27m43.803002346s) [upload=474834]: Upload finished. err: <nil> Duration: 30.002182152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX......] duration: 4.937736562s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:57:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.937767131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.....] duration: 5.001493743s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:56:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X......X................................................X.] duration: 50.027071675s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 04:56:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 50.02709491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.526283975s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:56:30 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.526309817s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XX] duration: 30.002981257s trace: https://jaeger.dev.storj.io/trace/6252e020c7ae4d89 FAIL (2025-12-28 04:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 7NZAVKMTTO6BHQHGDDDLQWKD2LVPRKQFOANHITDQ55HTBJ3DCPNA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: C5MX4JEHM5OBAXLZ4UKHG7UOXIGYK7QC6VYI6X326JC2J52JERUQ) context deadline exceeded ======LOGS====== alse Dec 28 04:55:53.298917644 (541h26m53.170000679s) [upload=474824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:55:53.298922754 (541h26m53.170005789s) [upload=474824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:55:53.546364239 (541h26m53.417447294s) [seg_pos={0 0}, upload=474824]: Begin upload segment... Dec 28 04:55:53.546435535 (541h26m53.41751858s) [seg_pos={0 0}, upload=474824]: Done begin upload segment. Dec 28 04:55:53.546496060 (541h26m53.417579105s) [noise=true , address=145.79.227.45:4229 , piece_id=5LJHLM7N2IY5BKSIL73HKDXJE7WZICPTAPIFH3KLITG7EQVFHAJQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.546526308 (541h26m53.417609353s) [noise=true , address=145.79.227.17:4970 , piece_id=HOAB5SDSPXOJW3UIQBXV3UFOYB4YTYBIIFVQT2RY4YSXHVZ4BHNA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.547393028 (541h26m53.418476084s) [noise=true , address=145.79.227.18:4007 , piece_id=RH3FL4JEF6XCSRNI2CISU6ZM5UP6H7TBURNYW4UA2E3JUIIVUM2A, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.547558232 (541h26m53.418641267s) [noise=true , address=145.79.227.48:4303 , piece_id=7A4W64TA2SL7Y3MW5DLONROFDTV7VKH27JDLX2LJSZZV6GPAZ7AQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.547963595 (541h26m53.41904663s) [noise=true , address=45.91.116.11:4118 , piece_id=HK76BTBHT3VNXG4UH2JJS3GUAXUYVURU2BCPFN6QLDCCQWKLH6LQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.548336084 (541h26m53.419419119s) [noise=true , address=45.91.116.7:4012 , piece_id=Y7DOVLHMHJN435DEVNNDYUM5HUC7WVF4SZVK5SQ7LZ2P4LTYYK4Q, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.548728531 (541h26m53.419811566s) [noise=true , address=45.91.116.12:4226 , piece_id=4DNQK7JB4LFFUQNO5ZPE3KKFJRXD66YS5GRTFVDZMYV7IAVSHVMQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.550483803 (541h26m53.421566838s) [noise=true , address=45.91.116.9:4261 , piece_id=NVQX6UYVAXLM24RFJSGZAE532UI2WXSWFK3ZWAFBGUYRVSUC6H3Q, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.551201560 (541h26m53.422284595s) [noise=true , address=145.79.227.46:4602 , piece_id=NCWARPAC6X7WTLGNY6GBW7MCVZVY3LNYYMPIVB52LSY4BSHQOTRA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.552381968 (541h26m53.423465003s) [noise=true , address=145.79.227.47:4267 , piece_id=G5DQYRMN2ATKS7FFARBYYQ7K45K2FIKDZ5AWZWKFQU2ALGKVWUVA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.553527800 (541h26m53.424610845s) [noise=true , address=45.91.116.13:4154 , piece_id=7NZAVKMTTO6BHQHGDDDLQWKD2LVPRKQFOANHITDQ55HTBJ3DCPNA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.554933717 (541h26m53.426016752s) [noise=true , address=77.90.31.14:4662 , piece_id=C5MX4JEHM5OBAXLZ4UKHG7UOXIGYK7QC6VYI6X326JC2J52JERUQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.556069299 (541h26m53.427152334s) [noise=true , address=45.91.116.10:4083 , piece_id=BABNZEZXRQIXYAPEZEQGHHHWSQWHVEE4PBP6BKG3BW6RFNOYJBRA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.557465778 (541h26m53.428548813s) [noise=true , address=145.79.227.51:4409 , piece_id=FO4PQ7U3NZ7UHUFXN3XXKJBRRLJKNEXTLHRBJWDH2263EUE23MVA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.559189119 (541h26m53.430272164s) [noise=true , address=145.79.227.14:4840 , piece_id=GYLS42B3O34UKR2TUI2XXDA7HKPXZCCFSFLSKIM3MLXV6MKWDLSQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.560334921 (541h26m53.431417966s) [noise=true , address=145.79.227.24:4583 , piece_id=3HEUI5DJQG3LYSPOHJWMBHCAVEH7GXQPQURP53FTSBWOCR47C3SA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:55.233212892 (541h26m55.104295928s) [upload=474824]: Next returned nil segment Dec 28 04:55:55.233232870 (541h26m55.104315915s) [upload=474824]: Waiting for error group managing segments... Dec 28 04:55:55.556686274 (541h26m55.427769319s) [noise=true , address=145.79.227.17:4970 , piece_id=HOAB5SDSPXOJW3UIQBXV3UFOYB4YTYBIIFVQT2RY4YSXHVZ4BHNA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.558080840 (541h26m55.429163885s) [noise=true , address=145.79.227.46:4602 , piece_id=NCWARPAC6X7WTLGNY6GBW7MCVZVY3LNYYMPIVB52LSY4BSHQOTRA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.559563834 (541h26m55.430646869s) [noise=true , address=145.79.227.45:4229 , piece_id=5LJHLM7N2IY5BKSIL73HKDXJE7WZICPTAPIFH3KLITG7EQVFHAJQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.563317130 (541h26m55.434400175s) [noise=true , address=145.79.227.18:4007 , piece_id=RH3FL4JEF6XCSRNI2CISU6ZM5UP6H7TBURNYW4UA2E3JUIIVUM2A, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.567219010 (541h26m55.438302045s) [noise=true , address=145.79.227.48:4303 , piece_id=7A4W64TA2SL7Y3MW5DLONROFDTV7VKH27JDLX2LJSZZV6GPAZ7AQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.575212431 (541h26m55.446295466s) [noise=true , address=145.79.227.47:4267 , piece_id=G5DQYRMN2ATKS7FFARBYYQ7K45K2FIKDZ5AWZWKFQU2ALGKVWUVA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.577744402 (541h26m55.448827437s) [noise=true , address=145.79.227.14:4840 , piece_id=GYLS42B3O34UKR2TUI2XXDA7HKPXZCCFSFLSKIM3MLXV6MKWDLSQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.583893000 (541h26m55.454976045s) [noise=true , address=145.79.227.24:4583 , piece_id=3HEUI5DJQG3LYSPOHJWMBHCAVEH7GXQPQURP53FTSBWOCR47C3SA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.593375135 (541h26m55.46445818s) [noise=true , address=145.79.227.51:4409 , piece_id=FO4PQ7U3NZ7UHUFXN3XXKJBRRLJKNEXTLHRBJWDH2263EUE23MVA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.835459841 (541h26m55.706542876s) [noise=true , address=77.90.31.14:4662 , piece_id=C5MX4JEHM5OBAXLZ4UKHG7UOXIGYK7QC6VYI6X326JC2J52JERUQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.854465240 (541h26m55.725548275s) [noise=true , address=45.91.116.12:4226 , piece_id=4DNQK7JB4LFFUQNO5ZPE3KKFJRXD66YS5GRTFVDZMYV7IAVSHVMQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.890042224 (541h26m55.761125269s) [noise=true , address=45.91.116.9:4261 , piece_id=NVQX6UYVAXLM24RFJSGZAE532UI2WXSWFK3ZWAFBGUYRVSUC6H3Q, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.923662743 (541h26m55.794745788s) [noise=true , address=45.91.116.13:4154 , piece_id=7NZAVKMTTO6BHQHGDDDLQWKD2LVPRKQFOANHITDQ55HTBJ3DCPNA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.923687490 (541h26m55.794770545s) [seg_pos={0 0}, upload=474824]: Segment reached optimal threshold of 13 pieces. Dec 28 04:55:55.923958245 (541h26m55.79504129s) [noise=true , address=45.91.116.11:4118 , piece_id=HK76BTBHT3VNXG4UH2JJS3GUAXUYVURU2BCPFN6QLDCCQWKLH6LQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:57286->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:55:55.923968545 (541h26m55.79505159s) [noise=true , address=45.91.116.11:4118 , piece_id=HK76BTBHT3VNXG4UH2JJS3GUAXUYVURU2BCPFN6QLDCCQWKLH6LQ, seg_pos={0 0}, upload=474824]: Long tail cancellation due to optimal threshold Dec 28 04:55:55.923979085 (541h26m55.79506212s) [noise=true , address=45.91.116.7:4012 , piece_id=Y7DOVLHMHJN435DEVNNDYUM5HUC7WVF4SZVK5SQ7LZ2P4LTYYK4Q, seg_pos={0 0}, upload=474824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:60390->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:60390->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60390->45.91.116.7:4012: use of closed network connection Dec 28 04:55:55.923986890 (541h26m55.795069935s) [noise=true , address=45.91.116.7:4012 , piece_id=Y7DOVLHMHJN435DEVNNDYUM5HUC7WVF4SZVK5SQ7LZ2P4LTYYK4Q, seg_pos={0 0}, upload=474824]: Long tail cancellation due to optimal threshold Dec 28 04:55:55.924119062 (541h26m55.795202107s) [noise=true , address=45.91.116.10:4083 , piece_id=BABNZEZXRQIXYAPEZEQGHHHWSQWHVEE4PBP6BKG3BW6RFNOYJBRA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:49262->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:49262->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49262->45.91.116.10:4083: use of closed network connection Dec 28 04:55:55.924127517 (541h26m55.795210553s) [noise=true , address=45.91.116.10:4083 , piece_id=BABNZEZXRQIXYAPEZEQGHHHWSQWHVEE4PBP6BKG3BW6RFNOYJBRA, seg_pos={0 0}, upload=474824]: Long tail cancellation due to optimal threshold Dec 28 04:55:55.924231075 (541h26m55.79531411s) [seg_pos={0 0}, upload=474824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:55:55.924317610 (541h26m55.795400655s) [upload=474824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:55:55.924324413 (541h26m55.795407448s) [upload=474824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:55:56.212721053 (541h26m56.083804098s) [upload=474824]: Done uploading segments... Dec 28 04:55:56.212740581 (541h26m56.083823616s) [upload=474824]: Upload finished. err: <nil> Duration: 30.00300423s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX....] duration: 3.491236959s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:56:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491266995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.876117013s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:55:51 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 7.876141458s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X......X................................................X] duration: 1m0.004690157s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 04:55:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 1m0.004703944s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX...] duration: 4.012328065s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:55:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.012379744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.X] duration: 30.250510569s trace: https://jaeger.dev.storj.io/trace/86cd76882404dbf FAIL (2025-12-28 04:55:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 28 04:54:49.653453884 (541h25m49.524536929s) [upload=474812]: Starting upload Dec 28 04:54:49.653597267 (541h25m49.524680312s) [upload=474812]: Done starting upload Dec 28 04:54:49.653658754 (541h25m49.524741799s) [upload=474812]: Uploading segments... Dec 28 04:54:49.654014451 (541h25m49.525097496s) [upload=474812]: Got next segment. Inline: false Dec 28 04:54:49.654076109 (541h25m49.525159154s) [upload=474812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:54:49.654082230 (541h25m49.525165275s) [upload=474812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:54:49.897536083 (541h25m49.768619128s) [seg_pos={0 0}, upload=474812]: Begin upload segment... Dec 28 04:54:49.897600455 (541h25m49.76868351s) [seg_pos={0 0}, upload=474812]: Done begin upload segment. Dec 28 04:54:49.897690196 (541h25m49.768773241s) [noise=true , address=77.90.31.15:4697 , piece_id=MXGUEVP3I7A6POAKNFCE6FGGVJ22W3X6HJAZ5FMEJREEJ6UMER5A, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.897699544 (541h25m49.768782589s) [noise=true , address=145.79.227.38:4977 , piece_id=VGWXTQUT3M5JJDTKJRQBXEDDQCBH5LL6SAVBBQOBADDXEKJHVKLA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.898899679 (541h25m49.769982724s) [noise=true , address=145.79.227.10:4403 , piece_id=J4BAJTPKS4Q35DNDKQ3AKL3PCAQMNP3CV6DWVGYNK3TKISR3O52A, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.899491887 (541h25m49.770574922s) [noise=true , address=145.79.227.48:4303 , piece_id=VRW6US5GNPTQAJP5CWPXRKUP234ULZ4SJHSJHILZYFZTKIM3KU6A, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.900689968 (541h25m49.771773013s) [noise=true , address=145.79.227.12:4765 , piece_id=QLSSBYEI76XZQHJBHMNFVRRRU4G234NZGSE23DZ3OORXEOEKGHGQ, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.901465465 (541h25m49.77254851s) [noise=true , address=145.79.227.37:4566 , piece_id=WULR2SAELC7UFQWDRHTQ7RD4QV2TWF2IWV6JR3LRJUCOJPDJLMPA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.903223973 (541h25m49.774307008s) [noise=true , address=145.79.227.51:4410 , piece_id=JHAO2QMP6UU4GBSYEGZOB67AXMMYL33ISL57RQLK4I7CQ54FKTWA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.903839766 (541h25m49.774922811s) [noise=true , address=145.79.227.46:4601 , piece_id=FFE47R5Z57XU56I2CE7DOO4DGJLEZ3T2GMSQA65UT67BU6IPBH3Q, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.903907024 (541h25m49.774990069s) [noise=true , address=145.79.227.29:4941 , piece_id=OKZVRGHADKX2TM6T7LX6WJM7NLKMRGYZLHAWAHU2HQMMMH2UOTMA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.905606039 (541h25m49.776689084s) [noise=true , address=145.79.227.45:4231 , piece_id=NG7VBSJ56VPOKU7ND54LHLKY6NZSSX6KXQUKPM2FUI4Q7MH3S34Q, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.906923738 (541h25m49.778006783s) [noise=true , address=145.79.227.5:4221 , piece_id=X4CCIJM4WBTPRD5VV5DY3R6HLVH3KVEZBHOR5UKCZU2J5F7ZA24Q, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.909545309 (541h25m49.780628354s) [noise=true , address=77.90.31.14:4664 , piece_id=ZHQWEUCQEKKKJBC7TOS6SNDXFHMJPZ3WR76O6ZRI4JBYDFAFYOZQ, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.911802538 (541h25m49.782885583s) [noise=true , address=77.90.31.19:4844 , piece_id=MHW7DAB6Q6QNXHHTCYN2LKGNX7OTLIIIOTC5KJB6VC4AAER7GULQ, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.914125171 (541h25m49.785208206s) [noise=true , address=145.79.227.49:4337 , piece_id=LXZMFTAQHBRHVEK3C4ZBHXI4PEZDY3I4AFZJY223MUJ2HRH7A2OA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.918405471 (541h25m49.789488516s) [noise=true , address=145.79.227.52:4447 , piece_id=EVLCICHGDCSN5GNKMNVNGEVD4PY6QOU2B6D7I5EH6EC5AOG2XWZA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.919805717 (541h25m49.790888752s) [noise=true , address=45.91.116.4:4939 , piece_id=IJTT33HWOA3VUKPQNRWXPBQNF6OPECF75RHJ5JI7PKFJXQAP7PFA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:51.787343730 (541h25m51.658426775s) [upload=474812]: Next returned nil segment Dec 28 04:54:51.787359349 (541h25m51.658442394s) [upload=474812]: Waiting for error group managing segments... Dec 28 04:54:52.087775788 (541h25m51.958858833s) [noise=true , address=77.90.31.15:4697 , piece_id=MXGUEVP3I7A6POAKNFCE6FGGVJ22W3X6HJAZ5FMEJREEJ6UMER5A, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:52.429143603 (541h25m52.300226638s) [noise=true , address=77.90.31.19:4844 , piece_id=MHW7DAB6Q6QNXHHTCYN2LKGNX7OTLIIIOTC5KJB6VC4AAER7GULQ, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:52.926529520 (541h25m52.797612555s) [noise=true , address=145.79.227.5:4221 , piece_id=X4CCIJM4WBTPRD5VV5DY3R6HLVH3KVEZBHOR5UKCZU2J5F7ZA24Q, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:53.048402815 (541h25m52.91948586s) [noise=true , address=45.91.116.4:4939 , piece_id=IJTT33HWOA3VUKPQNRWXPBQNF6OPECF75RHJ5JI7PKFJXQAP7PFA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:53.201991179 (541h25m53.073074214s) [noise=true , address=77.90.31.14:4664 , piece_id=ZHQWEUCQEKKKJBC7TOS6SNDXFHMJPZ3WR76O6ZRI4JBYDFAFYOZQ, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:54.036061573 (541h25m53.907144608s) [noise=true , address=145.79.227.49:4337 , piece_id=LXZMFTAQHBRHVEK3C4ZBHXI4PEZDY3I4AFZJY223MUJ2HRH7A2OA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:54.110997712 (541h25m53.982080757s) [noise=true , address=145.79.227.37:4566 , piece_id=WULR2SAELC7UFQWDRHTQ7RD4QV2TWF2IWV6JR3LRJUCOJPDJLMPA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:54.634168397 (541h25m54.505251432s) [noise=true , address=145.79.227.51:4410 , piece_id=JHAO2QMP6UU4GBSYEGZOB67AXMMYL33ISL57RQLK4I7CQ54FKTWA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:54.844988341 (541h25m54.716071386s) [noise=true , address=145.79.227.12:4765 , piece_id=QLSSBYEI76XZQHJBHMNFVRRRU4G234NZGSE23DZ3OORXEOEKGHGQ, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:58.331306556 (541h25m58.202389591s) [noise=true , address=145.79.227.29:4941 , piece_id=OKZVRGHADKX2TM6T7LX6WJM7NLKMRGYZLHAWAHU2HQMMMH2UOTMA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:55:07.710412450 (541h26m7.581495495s) [noise=true , address=145.79.227.10:4403 , piece_id=J4BAJTPKS4Q35DNDKQ3AKL3PCAQMNP3CV6DWVGYNK3TKISR3O52A, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:55:15.123476617 (541h26m14.994559662s) [noise=true , address=145.79.227.48:4303 , piece_id=VRW6US5GNPTQAJP5CWPXRKUP234ULZ4SJHSJHILZYFZTKIM3KU6A, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:55:18.971916354 (541h26m18.842999389s) [noise=true , address=145.79.227.52:4447 , piece_id=EVLCICHGDCSN5GNKMNVNGEVD4PY6QOU2B6D7I5EH6EC5AOG2XWZA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: read tcp 188.166.225.170:41730->145.79.227.52:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:55:18.972105415 (541h26m18.84318846s) [noise=true , address=145.79.227.45:4231 , piece_id=NG7VBSJ56VPOKU7ND54LHLKY6NZSSX6KXQUKPM2FUI4Q7MH3S34Q, seg_pos={0 0}, upload=474812]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:145.79.227.45:4231): protocol: read tcp 188.166.225.170:52538->145.79.227.45:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:55:18.972198081 (541h26m18.843281256s) [noise=true , address=145.79.227.46:4601 , piece_id=FFE47R5Z57XU56I2CE7DOO4DGJLEZ3T2GMSQA65UT67BU6IPBH3Q, seg_pos={0 0}, upload=474812]: Done uploading piece. err: ecclient: upload failed (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, address:145.79.227.46:4601): protocol: expected piece hash; context deadline exceeded; EOF Dec 28 04:55:18.972615295 (541h26m18.84369833s) [noise=true , address=145.79.227.38:4977 , piece_id=VGWXTQUT3M5JJDTKJRQBXEDDQCBH5LL6SAVBBQOBADDXEKJHVKLA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: context deadline exceeded; context deadline exceeded Dec 28 04:55:18.972682253 (541h26m18.843765288s) [seg_pos={0 0}, upload=474812]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 28 04:55:19.220591810 (541h26m19.091674845s) [upload=474812]: Done uploading segments... Dec 28 04:55:19.220624301 (541h26m19.091707347s) [upload=474812]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.250531239s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.005959058s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:55:13 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 30.005979526s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......X................................................] duration: 42.010403922s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:54:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 42.010423951s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX..] duration: 3.943653218s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:54:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.943681791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.] duration: 6.439570213s trace: https://jaeger.dev.storj.io/trace/701d7ac209094715 FAIL (2025-12-28 04:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:52:57.918408250 (541h23m57.789491285s) [upload=474784]: Starting upload Dec 28 04:52:57.918473895 (541h23m57.789556931s) [upload=474784]: Done starting upload Dec 28 04:52:57.918783044 (541h23m57.789866089s) [upload=474784]: Uploading segments... Dec 28 04:52:57.918869348 (541h23m57.789952383s) [upload=474784]: Got next segment. Inline: false Dec 28 04:52:57.918900868 (541h23m57.789983913s) [upload=474784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:52:57.918905397 (541h23m57.789988432s) [upload=474784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:52:58.165602625 (541h23m58.03668567s) [seg_pos={0 0}, upload=474784]: Begin upload segment... Dec 28 04:52:58.165699410 (541h23m58.036782455s) [seg_pos={0 0}, upload=474784]: Done begin upload segment. Dec 28 04:52:58.165745998 (541h23m58.036829083s) [noise=true , address=145.79.227.15:4874 , piece_id=CVQXNQVDLL377EH43H2XNQVUJWO3QM4XCBURHVWFA5NGU7NTQUIQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.165913176 (541h23m58.036996222s) [noise=true , address=45.91.116.10:4082 , piece_id=DNDRI6R325BARR3OW67PEWSCUGYPH3VMHCQ45QPDYZPXL7WEZPIQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.166774156 (541h23m58.037857191s) [noise=true , address=145.79.227.5:4222 , piece_id=4GQO4AW2HW5TAETKCZLR4P6ZEPQLEA4YQOXOQWDCPFJ4ESB4MP3A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.166816276 (541h23m58.037899311s) [noise=true , address=145.79.227.35:4482 , piece_id=VJKDZMJS3KRUMOM6BDX545FXB5GM4F65YKVUSLKVQ35ZGGQ4FPYA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167248920 (541h23m58.038331985s) [noise=true , address=145.79.227.6:4257 , piece_id=FRFKUD7RZXCEMMA73TFBYMU7QCQWNGBUPVUY6HK23AYCQZ6JHUOQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167392754 (541h23m58.038475789s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167617742 (541h23m58.038700787s) [noise=true , address=145.79.227.36:4519 , piece_id=ER4VFPHO3QUAM3JZPU4ZULUM2RRO2HU3YSGVABNVDKQU6T2KK5EQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167955335 (541h23m58.03903839s) [noise=true , address=77.90.31.19:4844 , piece_id=WNDL74JK4RATDVSGHJWHFLJD4E4LSRI3V7G7YYSAVRCZFSDDEV6A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167969201 (541h23m58.039052236s) [noise=true , address=145.79.227.39:4015 , piece_id=GOQENX5CEY3TKIHJYDZJTNZ6VIRMX4B3LRXYVOMEDPKPQJ6VXFEA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168239336 (541h23m58.039322371s) [noise=true , address=145.79.227.37:4567 , piece_id=Q56OJOMH2WZLGVWYAKXG45QF4HQXSTFQNYM3VF5W46LQR2E6Y5QA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168571358 (541h23m58.039654393s) [noise=true , address=45.91.116.4:4937 , piece_id=MN43HKOVCGZTY2EM2QCLSFP4LO4S7TMYJ7PLBNANMMZON7WZNC2A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168598650 (541h23m58.039681695s) [noise=true , address=145.79.227.42:4122 , piece_id=MWMYEVTN6YKU36IWKQ7TVIH25BN4CMMZGMMYI3I4OTWUZOYHEWBA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168841212 (541h23m58.039924247s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.169174677 (541h23m58.040257712s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.169827650 (541h23m58.040910695s) [noise=true , address=145.79.227.13:4802 , piece_id=TPGHRJTLR3TETNDIEDFGQ76FGPXNN4DE5JOPJDMV645QK5SFEB6A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.170489139 (541h23m58.041572174s) [noise=true , address=77.90.31.21:4877 , piece_id=RSRSE2P6UKF3BIDZK7DIGWG6ZPVTDZZEYHW5A5F6D6SWBFUJX7EQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:59.885866567 (541h23m59.756949602s) [upload=474784]: Next returned nil segment Dec 28 04:52:59.885889631 (541h23m59.756972686s) [upload=474784]: Waiting for error group managing segments... Dec 28 04:53:00.185504883 (541h24m0.056587929s) [noise=true , address=145.79.227.15:4874 , piece_id=CVQXNQVDLL377EH43H2XNQVUJWO3QM4XCBURHVWFA5NGU7NTQUIQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.185687903 (541h24m0.056770938s) [noise=true , address=145.79.227.39:4015 , piece_id=GOQENX5CEY3TKIHJYDZJTNZ6VIRMX4B3LRXYVOMEDPKPQJ6VXFEA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.189399379 (541h24m0.060482414s) [noise=true , address=145.79.227.42:4122 , piece_id=MWMYEVTN6YKU36IWKQ7TVIH25BN4CMMZGMMYI3I4OTWUZOYHEWBA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.190123118 (541h24m0.061206153s) [noise=true , address=145.79.227.35:4482 , piece_id=VJKDZMJS3KRUMOM6BDX545FXB5GM4F65YKVUSLKVQ35ZGGQ4FPYA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.202143737 (541h24m0.073226772s) [noise=true , address=145.79.227.13:4802 , piece_id=TPGHRJTLR3TETNDIEDFGQ76FGPXNN4DE5JOPJDMV645QK5SFEB6A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.202523209 (541h24m0.073606254s) [noise=true , address=145.79.227.37:4567 , piece_id=Q56OJOMH2WZLGVWYAKXG45QF4HQXSTFQNYM3VF5W46LQR2E6Y5QA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.211302837 (541h24m0.082385872s) [noise=true , address=145.79.227.36:4519 , piece_id=ER4VFPHO3QUAM3JZPU4ZULUM2RRO2HU3YSGVABNVDKQU6T2KK5EQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.217272053 (541h24m0.088355098s) [noise=true , address=145.79.227.6:4257 , piece_id=FRFKUD7RZXCEMMA73TFBYMU7QCQWNGBUPVUY6HK23AYCQZ6JHUOQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.433488362 (541h24m0.304571407s) [noise=true , address=45.91.116.10:4082 , piece_id=DNDRI6R325BARR3OW67PEWSCUGYPH3VMHCQ45QPDYZPXL7WEZPIQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.468101320 (541h24m0.339184365s) [noise=true , address=77.90.31.21:4877 , piece_id=RSRSE2P6UKF3BIDZK7DIGWG6ZPVTDZZEYHW5A5F6D6SWBFUJX7EQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.490711994 (541h24m0.361795039s) [noise=true , address=77.90.31.19:4844 , piece_id=WNDL74JK4RATDVSGHJWHFLJD4E4LSRI3V7G7YYSAVRCZFSDDEV6A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.556360428 (541h24m0.427443473s) [noise=true , address=145.79.227.5:4222 , piece_id=4GQO4AW2HW5TAETKCZLR4P6ZEPQLEA4YQOXOQWDCPFJ4ESB4MP3A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.570783703 (541h24m0.441866738s) [noise=true , address=45.91.116.4:4937 , piece_id=MN43HKOVCGZTY2EM2QCLSFP4LO4S7TMYJ7PLBNANMMZON7WZNC2A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.570803741 (541h24m0.441886776s) [seg_pos={0 0}, upload=474784]: Segment reached optimal threshold of 13 pieces. Dec 28 04:53:00.570993462 (541h24m0.442076538s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 04:53:00.571003722 (541h24m0.442086787s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571152184 (541h24m0.442235219s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection Dec 28 04:53:00.571158215 (541h24m0.44224125s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571485469 (541h24m0.442568524s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:53:00.571506499 (541h24m0.442589544s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571576261 (541h24m0.442659307s) [seg_pos={0 0}, upload=474784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:53:00.571688685 (541h24m0.44277173s) [upload=474784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:53:00.571697673 (541h24m0.442780708s) [upload=474784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:53:00.861235446 (541h24m0.732318491s) [upload=474784]: Done uploading segments... Dec 28 04:53:00.861273549 (541h24m0.732356594s) [upload=474784]: Upload finished. err: <nil> Duration: 6.439637372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......X...............................................] duration: 59.051729003s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 59.051757546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX] duration: 30.004037924s trace: https://jaeger.dev.storj.io/trace/701d7ac209094715 FAIL (2025-12-28 04:53:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:52:57.918408250 (541h23m57.789491285s) [upload=474784]: Starting upload Dec 28 04:52:57.918473895 (541h23m57.789556931s) [upload=474784]: Done starting upload Dec 28 04:52:57.918783044 (541h23m57.789866089s) [upload=474784]: Uploading segments... Dec 28 04:52:57.918869348 (541h23m57.789952383s) [upload=474784]: Got next segment. Inline: false Dec 28 04:52:57.918900868 (541h23m57.789983913s) [upload=474784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:52:57.918905397 (541h23m57.789988432s) [upload=474784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:52:58.165602625 (541h23m58.03668567s) [seg_pos={0 0}, upload=474784]: Begin upload segment... Dec 28 04:52:58.165699410 (541h23m58.036782455s) [seg_pos={0 0}, upload=474784]: Done begin upload segment. Dec 28 04:52:58.165745998 (541h23m58.036829083s) [noise=true , address=145.79.227.15:4874 , piece_id=CVQXNQVDLL377EH43H2XNQVUJWO3QM4XCBURHVWFA5NGU7NTQUIQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.165913176 (541h23m58.036996222s) [noise=true , address=45.91.116.10:4082 , piece_id=DNDRI6R325BARR3OW67PEWSCUGYPH3VMHCQ45QPDYZPXL7WEZPIQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.166774156 (541h23m58.037857191s) [noise=true , address=145.79.227.5:4222 , piece_id=4GQO4AW2HW5TAETKCZLR4P6ZEPQLEA4YQOXOQWDCPFJ4ESB4MP3A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.166816276 (541h23m58.037899311s) [noise=true , address=145.79.227.35:4482 , piece_id=VJKDZMJS3KRUMOM6BDX545FXB5GM4F65YKVUSLKVQ35ZGGQ4FPYA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167248920 (541h23m58.038331985s) [noise=true , address=145.79.227.6:4257 , piece_id=FRFKUD7RZXCEMMA73TFBYMU7QCQWNGBUPVUY6HK23AYCQZ6JHUOQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167392754 (541h23m58.038475789s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167617742 (541h23m58.038700787s) [noise=true , address=145.79.227.36:4519 , piece_id=ER4VFPHO3QUAM3JZPU4ZULUM2RRO2HU3YSGVABNVDKQU6T2KK5EQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167955335 (541h23m58.03903839s) [noise=true , address=77.90.31.19:4844 , piece_id=WNDL74JK4RATDVSGHJWHFLJD4E4LSRI3V7G7YYSAVRCZFSDDEV6A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167969201 (541h23m58.039052236s) [noise=true , address=145.79.227.39:4015 , piece_id=GOQENX5CEY3TKIHJYDZJTNZ6VIRMX4B3LRXYVOMEDPKPQJ6VXFEA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168239336 (541h23m58.039322371s) [noise=true , address=145.79.227.37:4567 , piece_id=Q56OJOMH2WZLGVWYAKXG45QF4HQXSTFQNYM3VF5W46LQR2E6Y5QA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168571358 (541h23m58.039654393s) [noise=true , address=45.91.116.4:4937 , piece_id=MN43HKOVCGZTY2EM2QCLSFP4LO4S7TMYJ7PLBNANMMZON7WZNC2A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168598650 (541h23m58.039681695s) [noise=true , address=145.79.227.42:4122 , piece_id=MWMYEVTN6YKU36IWKQ7TVIH25BN4CMMZGMMYI3I4OTWUZOYHEWBA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168841212 (541h23m58.039924247s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.169174677 (541h23m58.040257712s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.169827650 (541h23m58.040910695s) [noise=true , address=145.79.227.13:4802 , piece_id=TPGHRJTLR3TETNDIEDFGQ76FGPXNN4DE5JOPJDMV645QK5SFEB6A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.170489139 (541h23m58.041572174s) [noise=true , address=77.90.31.21:4877 , piece_id=RSRSE2P6UKF3BIDZK7DIGWG6ZPVTDZZEYHW5A5F6D6SWBFUJX7EQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:59.885866567 (541h23m59.756949602s) [upload=474784]: Next returned nil segment Dec 28 04:52:59.885889631 (541h23m59.756972686s) [upload=474784]: Waiting for error group managing segments... Dec 28 04:53:00.185504883 (541h24m0.056587929s) [noise=true , address=145.79.227.15:4874 , piece_id=CVQXNQVDLL377EH43H2XNQVUJWO3QM4XCBURHVWFA5NGU7NTQUIQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.185687903 (541h24m0.056770938s) [noise=true , address=145.79.227.39:4015 , piece_id=GOQENX5CEY3TKIHJYDZJTNZ6VIRMX4B3LRXYVOMEDPKPQJ6VXFEA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.189399379 (541h24m0.060482414s) [noise=true , address=145.79.227.42:4122 , piece_id=MWMYEVTN6YKU36IWKQ7TVIH25BN4CMMZGMMYI3I4OTWUZOYHEWBA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.190123118 (541h24m0.061206153s) [noise=true , address=145.79.227.35:4482 , piece_id=VJKDZMJS3KRUMOM6BDX545FXB5GM4F65YKVUSLKVQ35ZGGQ4FPYA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.202143737 (541h24m0.073226772s) [noise=true , address=145.79.227.13:4802 , piece_id=TPGHRJTLR3TETNDIEDFGQ76FGPXNN4DE5JOPJDMV645QK5SFEB6A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.202523209 (541h24m0.073606254s) [noise=true , address=145.79.227.37:4567 , piece_id=Q56OJOMH2WZLGVWYAKXG45QF4HQXSTFQNYM3VF5W46LQR2E6Y5QA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.211302837 (541h24m0.082385872s) [noise=true , address=145.79.227.36:4519 , piece_id=ER4VFPHO3QUAM3JZPU4ZULUM2RRO2HU3YSGVABNVDKQU6T2KK5EQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.217272053 (541h24m0.088355098s) [noise=true , address=145.79.227.6:4257 , piece_id=FRFKUD7RZXCEMMA73TFBYMU7QCQWNGBUPVUY6HK23AYCQZ6JHUOQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.433488362 (541h24m0.304571407s) [noise=true , address=45.91.116.10:4082 , piece_id=DNDRI6R325BARR3OW67PEWSCUGYPH3VMHCQ45QPDYZPXL7WEZPIQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.468101320 (541h24m0.339184365s) [noise=true , address=77.90.31.21:4877 , piece_id=RSRSE2P6UKF3BIDZK7DIGWG6ZPVTDZZEYHW5A5F6D6SWBFUJX7EQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.490711994 (541h24m0.361795039s) [noise=true , address=77.90.31.19:4844 , piece_id=WNDL74JK4RATDVSGHJWHFLJD4E4LSRI3V7G7YYSAVRCZFSDDEV6A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.556360428 (541h24m0.427443473s) [noise=true , address=145.79.227.5:4222 , piece_id=4GQO4AW2HW5TAETKCZLR4P6ZEPQLEA4YQOXOQWDCPFJ4ESB4MP3A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.570783703 (541h24m0.441866738s) [noise=true , address=45.91.116.4:4937 , piece_id=MN43HKOVCGZTY2EM2QCLSFP4LO4S7TMYJ7PLBNANMMZON7WZNC2A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.570803741 (541h24m0.441886776s) [seg_pos={0 0}, upload=474784]: Segment reached optimal threshold of 13 pieces. Dec 28 04:53:00.570993462 (541h24m0.442076538s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 04:53:00.571003722 (541h24m0.442086787s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571152184 (541h24m0.442235219s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection Dec 28 04:53:00.571158215 (541h24m0.44224125s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571485469 (541h24m0.442568524s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:53:00.571506499 (541h24m0.442589544s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571576261 (541h24m0.442659307s) [seg_pos={0 0}, upload=474784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:53:00.571688685 (541h24m0.44277173s) [upload=474784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:53:00.571697673 (541h24m0.442780708s) [upload=474784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:53:00.861235446 (541h24m0.732318491s) [upload=474784]: Done uploading segments... Dec 28 04:53:00.861273549 (541h24m0.732356594s) [upload=474784]: Upload finished. err: <nil> Duration: 30.004060879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.] duration: 3.965497977s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:53:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.965529818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......X..............................................] duration: 52.805478621s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:53:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 52.805538776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XX] duration: 30.002906212s trace: https://jaeger.dev.storj.io/trace/7fda46f580656790 FAIL (2025-12-28 04:52:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 04:51:47.327032037 (541h22m47.198115072s) [upload=474765]: Starting upload Dec 28 04:51:47.327088313 (541h22m47.198171358s) [upload=474765]: Done starting upload Dec 28 04:51:47.327155441 (541h22m47.198238486s) [upload=474765]: Uploading segments... Dec 28 04:51:47.327295948 (541h22m47.198378984s) [upload=474765]: Got next segment. Inline: false Dec 28 04:51:47.327324012 (541h22m47.198407057s) [upload=474765]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:51:47.327328711 (541h22m47.198411746s) [upload=474765]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:51:47.573755826 (541h22m47.444838861s) [seg_pos={0 0}, upload=474765]: Begin upload segment... Dec 28 04:51:47.573914758 (541h22m47.444997793s) [seg_pos={0 0}, upload=474765]: Done begin upload segment. Dec 28 04:51:47.574045497 (541h22m47.445128532s) [noise=true , address=77.90.31.14:4663 , piece_id=2WPA4CKLJMTVNXOGH4W6CWWRNB3BOZVVHTTIYIY2DD3RX7ODDVPQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.574337593 (541h22m47.445420628s) [noise=true , address=145.79.227.37:4565 , piece_id=AMHZRWGIYUIEHXBEJOZ7XCKISK5X3542AZAYCFWP2UMURJEUD63Q, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.575803004 (541h22m47.446886049s) [noise=true , address=145.79.227.18:4006 , piece_id=4QSY5ZJ65SMUA52X6UZYIXEAVBNAJ5M6FYD474AIPRWK5B2G5IZQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.575974821 (541h22m47.447057856s) [noise=true , address=45.91.116.11:4120 , piece_id=35FOAH3CSTPU324TR5FB2SUSGIM6KN253DZFA2ATF6454OHZGKTA, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.576260946 (541h22m47.447343991s) [noise=true , address=145.79.227.43:4159 , piece_id=AHQAKWHLZL7TBT5JJ3PRW36Q7AIBKF4355TZO72BIERO2ULJUPCQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.576338874 (541h22m47.447421909s) [noise=true , address=77.90.31.10:4518 , piece_id=TM5KZQEVHM4WUTXZSDMYNGW3MSQM7BWRMVDTI4M4A4XLLYID3QMQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.576554685 (541h22m47.44763772s) [noise=true , address=77.90.31.21:4878 , piece_id=6PT577EYUT7G65OCCKWAP4IJYOW6T26PQOG3PI36C6ZSRFP3BB5A, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.576619098 (541h22m47.447702133s) [noise=true , address=45.91.116.7:4009 , piece_id=43P7X3SPHDABSBPPQVAVFRQ3W2SJOMLGUX7GQUMPXL6OMWEOQ6FQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.576839537 (541h22m47.447922582s) [noise=true , address=45.91.116.14:4191 , piece_id=VA6YLUPGV6LKGPMZZD3KIWGYVHU4WK7XJUNUBKYHLNMG4IGKQSIA, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.576970907 (541h22m47.448053952s) [noise=true , address=145.79.227.14:4839 , piece_id=JL67LIOLAQGYYX3XLP7YO2LLZJ7RWQEYTGJQMIBIWPTEDSE4YUEQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.577203340 (541h22m47.448286375s) [noise=true , address=77.90.31.15:4697 , piece_id=DZF6C3J2EASBKBLHQ3L73FBBWKSGYRC5ZHZZKM26UHZB7FOHUNDQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.577379626 (541h22m47.448462661s) [noise=true , address=145.79.227.16:4912 , piece_id=3N3XSF2IRGB32PVARWVJW5BJI4X7VAL3DZ3ESGY3ICMLCHLST4QQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.577509994 (541h22m47.448593029s) [noise=true , address=145.79.227.31:4797 , piece_id=EJ6CFMOS5U742YOMVJKQSIUYP3EFK537CWPIX7MZNRNPAECFSR3A, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.577679045 (541h22m47.44876209s) [noise=true , address=145.79.227.46:4602 , piece_id=VCGJFRVQY5SV76TJIAO4ZZS43MXAWMP3F3LKY4QUUGCJHLPXJAXQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.578035405 (541h22m47.44911845s) [noise=true , address=145.79.227.41:4087 , piece_id=ORCXGCRSXXHJMTHL7WP7X2EEXK4K6PGCSX526T6ZSNTDZY6OH6KQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:47.578327010 (541h22m47.449410045s) [noise=true , address=145.79.227.52:4446 , piece_id=YNCXYCIPRBPB7LYQPXFRZKJQ3ONFHXAFB7WGC5DLMDFCED3VOFJQ, seg_pos={0 0}, upload=474765]: Uploading piece... Dec 28 04:51:49.308097934 (541h22m49.17918098s) [upload=474765]: Next returned nil segment Dec 28 04:51:49.308114756 (541h22m49.179197811s) [upload=474765]: Waiting for error group managing segments... Dec 28 04:51:49.588143843 (541h22m49.459226878s) [noise=true , address=145.79.227.18:4006 , piece_id=4QSY5ZJ65SMUA52X6UZYIXEAVBNAJ5M6FYD474AIPRWK5B2G5IZQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.588348944 (541h22m49.459431989s) [noise=true , address=145.79.227.16:4912 , piece_id=3N3XSF2IRGB32PVARWVJW5BJI4X7VAL3DZ3ESGY3ICMLCHLST4QQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.588682389 (541h22m49.459765434s) [noise=true , address=145.79.227.31:4797 , piece_id=EJ6CFMOS5U742YOMVJKQSIUYP3EFK537CWPIX7MZNRNPAECFSR3A, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.603459597 (541h22m49.474542632s) [noise=true , address=145.79.227.37:4565 , piece_id=AMHZRWGIYUIEHXBEJOZ7XCKISK5X3542AZAYCFWP2UMURJEUD63Q, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.611907864 (541h22m49.482990909s) [noise=true , address=145.79.227.52:4446 , piece_id=YNCXYCIPRBPB7LYQPXFRZKJQ3ONFHXAFB7WGC5DLMDFCED3VOFJQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.612228985 (541h22m49.48331202s) [noise=true , address=145.79.227.41:4087 , piece_id=ORCXGCRSXXHJMTHL7WP7X2EEXK4K6PGCSX526T6ZSNTDZY6OH6KQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.880029618 (541h22m49.751112714s) [noise=true , address=145.79.227.46:4602 , piece_id=VCGJFRVQY5SV76TJIAO4ZZS43MXAWMP3F3LKY4QUUGCJHLPXJAXQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.900531547 (541h22m49.771614592s) [noise=true , address=145.79.227.43:4159 , piece_id=AHQAKWHLZL7TBT5JJ3PRW36Q7AIBKF4355TZO72BIERO2ULJUPCQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.945318448 (541h22m49.816401493s) [noise=true , address=77.90.31.10:4518 , piece_id=TM5KZQEVHM4WUTXZSDMYNGW3MSQM7BWRMVDTI4M4A4XLLYID3QMQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:49.975314094 (541h22m49.846397139s) [noise=true , address=77.90.31.15:4697 , piece_id=DZF6C3J2EASBKBLHQ3L73FBBWKSGYRC5ZHZZKM26UHZB7FOHUNDQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:50.085630082 (541h22m49.956713117s) [noise=true , address=77.90.31.14:4663 , piece_id=2WPA4CKLJMTVNXOGH4W6CWWRNB3BOZVVHTTIYIY2DD3RX7ODDVPQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:50.492004570 (541h22m50.363087605s) [noise=true , address=45.91.116.7:4009 , piece_id=43P7X3SPHDABSBPPQVAVFRQ3W2SJOMLGUX7GQUMPXL6OMWEOQ6FQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:50.550706448 (541h22m50.421789483s) [noise=true , address=45.91.116.14:4191 , piece_id=VA6YLUPGV6LKGPMZZD3KIWGYVHU4WK7XJUNUBKYHLNMG4IGKQSIA, seg_pos={0 0}, upload=474765]: Done uploading piece. err: <nil> Dec 28 04:51:50.550722949 (541h22m50.421805985s) [seg_pos={0 0}, upload=474765]: Segment reached optimal threshold of 13 pieces. Dec 28 04:51:50.550888504 (541h22m50.421971539s) [noise=true , address=45.91.116.11:4120 , piece_id=35FOAH3CSTPU324TR5FB2SUSGIM6KN253DZFA2ATF6454OHZGKTA, seg_pos={0 0}, upload=474765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 04:51:50.550896760 (541h22m50.421979795s) [noise=true , address=45.91.116.11:4120 , piece_id=35FOAH3CSTPU324TR5FB2SUSGIM6KN253DZFA2ATF6454OHZGKTA, seg_pos={0 0}, upload=474765]: Long tail cancellation due to optimal threshold Dec 28 04:51:50.551220857 (541h22m50.422303892s) [noise=true , address=145.79.227.14:4839 , piece_id=JL67LIOLAQGYYX3XLP7YO2LLZJ7RWQEYTGJQMIBIWPTEDSE4YUEQ, seg_pos={0 0}, upload=474765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Dec 28 04:51:50.551229033 (541h22m50.422312068s) [noise=true , address=145.79.227.14:4839 , piece_id=JL67LIOLAQGYYX3XLP7YO2LLZJ7RWQEYTGJQMIBIWPTEDSE4YUEQ, seg_pos={0 0}, upload=474765]: Long tail cancellation due to optimal threshold Dec 28 04:51:50.551388176 (541h22m50.422471211s) [noise=true , address=77.90.31.21:4878 , piece_id=6PT577EYUT7G65OCCKWAP4IJYOW6T26PQOG3PI36C6ZSRFP3BB5A, seg_pos={0 0}, upload=474765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:55860->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:55860->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55860->77.90.31.21:4878: use of closed network connection Dec 28 04:51:50.551394257 (541h22m50.422477293s) [noise=true , address=77.90.31.21:4878 , piece_id=6PT577EYUT7G65OCCKWAP4IJYOW6T26PQOG3PI36C6ZSRFP3BB5A, seg_pos={0 0}, upload=474765]: Long tail cancellation due to optimal threshold Dec 28 04:51:50.551426799 (541h22m50.422509845s) [seg_pos={0 0}, upload=474765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:51:50.551521589 (541h22m50.422604624s) [upload=474765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:51:50.551527361 (541h22m50.422610396s) [upload=474765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:51:50.847151062 (541h22m50.718234097s) [upload=474765]: Done uploading segments... Dec 28 04:51:50.847179537 (541h22m50.718262572s) [upload=474765]: Upload finished. err: <nil> Duration: 30.002919357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX] duration: 5.000884333s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:52:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000907286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X......X.............................................] duration: 46.668066498s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:51:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 46.668083891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..X] duration: 30.00349505s trace: https://jaeger.dev.storj.io/trace/2a1cd0ed9743315a FAIL (2025-12-28 04:51:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: MDQ4WXQS2BH4X2QYWMXKWDTCINFD7DPBAIQ4ARPDHNFAKNBD4YRQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: PWLQC4BAHZFGDGSNVDDYXL332UFTP5KWOOGVOA4NY67MXQSXONTQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: QI7L7F4JNKFB4HYIAGML42ZVDB6EYUHUSVMAKE6MJALWITNYOYVQ) context deadline exceeded ======LOGS====== Dec 28 04:51:17.149242253 (541h22m17.020325288s) [upload=474760]: Starting upload Dec 28 04:51:17.149315222 (541h22m17.020398267s) [upload=474760]: Done starting upload Dec 28 04:51:17.149548957 (541h22m17.020631992s) [upload=474760]: Uploading segments... Dec 28 04:51:17.149597299 (541h22m17.020680334s) [upload=474760]: Got next segment. Inline: false Dec 28 04:51:17.149634129 (541h22m17.020717174s) [upload=474760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:51:17.149638778 (541h22m17.020721813s) [upload=474760]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:51:17.399569624 (541h22m17.270652659s) [seg_pos={0 0}, upload=474760]: Begin upload segment... Dec 28 04:51:17.399624398 (541h22m17.270707443s) [seg_pos={0 0}, upload=474760]: Done begin upload segment. Dec 28 04:51:17.399654736 (541h22m17.270737781s) [noise=true , address=45.91.116.12:4228 , piece_id=7RXBJQEKBCS6AJ24II3Z4IXPAPXCYEJH7Q2CPWSM36A4WS4WQ4RA, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.400689968 (541h22m17.271773013s) [noise=true , address=45.91.116.13:4155 , piece_id=4BWX5W47CA5WMIXK2F5GXERNGGAHPTXGUMNRBUEKAVHPWMJTGX6A, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.401211481 (541h22m17.272294526s) [noise=true , address=145.79.227.5:4221 , piece_id=VCO43DD4NWI3M2GVV57Z5QEENEAO5XBJDYVBRAVUZDQUNI244HUQ, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.401530568 (541h22m17.272613613s) [noise=true , address=145.79.227.16:4912 , piece_id=DW2G4Y7FJVSLQFWDFO27HPKUQBIYDW7NPRVCKWWKOW7JYHWAXCXQ, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.401856218 (541h22m17.272939253s) [noise=true , address=45.91.116.5:4297 , piece_id=WAILBHZLOCEHDLYTBT765SXSDMUN7TQSCGSEXHTU4A3UJ4LV7XMA, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.402163042 (541h22m17.273246077s) [noise=true , address=145.79.227.45:4229 , piece_id=HNSVMW4TQPW3FQSPGJK3W4LAKHMCEVXPB73ANRF4BGTIPISZ7TZA, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.404029146 (541h22m17.275112181s) [noise=true , address=145.79.227.27:4691 , piece_id=53DIZEGUEFJBCICGZNLXFQVVQBV2DAH2FEVYVR4CQLBLQRBYI5MA, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.404811255 (541h22m17.2758943s) [noise=true , address=45.91.116.10:4082 , piece_id=MDQ4WXQS2BH4X2QYWMXKWDTCINFD7DPBAIQ4ARPDHNFAKNBD4YRQ, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.406286173 (541h22m17.277369218s) [noise=true , address=45.91.116.4:4940 , piece_id=J2FJL5OH2RRBOYALWDXMI47C6PMD7DOEFFRVWVRZW5OKROCLFDJA, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.407835093 (541h22m17.278918138s) [noise=true , address=45.91.116.14:4190 , piece_id=PWLQC4BAHZFGDGSNVDDYXL332UFTP5KWOOGVOA4NY67MXQSXONTQ, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.409207476 (541h22m17.280290521s) [noise=true , address=145.79.227.29:4941 , piece_id=QGCD3N3AWTIBLH7A4WLW6Q4QOZFPVGTVGEEPLUTWCTY3MBDPRUGA, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.410861406 (541h22m17.281944451s) [noise=true , address=77.90.31.10:4520 , piece_id=QI7L7F4JNKFB4HYIAGML42ZVDB6EYUHUSVMAKE6MJALWITNYOYVQ, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.412120122 (541h22m17.283203167s) [noise=true , address=77.90.31.18:4808 , piece_id=IFXJH2X4AB4ZL5IOTNIYDSNCNW3WORSQKG2NGTSNC62BOZCQZCAA, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.412388444 (541h22m17.283471479s) [noise=true , address=145.79.227.37:4566 , piece_id=7ZOB5524IYMURSQTQHXNHSAKAKWMAWM5MD5ZP75VGY3D2G7KJUAQ, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.413828115 (541h22m17.28491115s) [noise=true , address=145.79.227.52:4447 , piece_id=7XNVVY4PTG7DMEQQ7642SQSCFKYYLGB74CNQTRIVVFM26PMWVJUQ, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:17.414992491 (541h22m17.286075536s) [noise=true , address=145.79.227.24:4581 , piece_id=YWWZSG4JIFVXNZPOEGGBANFMC67JWIEB4G4ZCII4MBVKR2E4F46A, seg_pos={0 0}, upload=474760]: Uploading piece... Dec 28 04:51:19.118793445 (541h22m18.98987649s) [upload=474760]: Next returned nil segment Dec 28 04:51:19.118836918 (541h22m18.989919963s) [upload=474760]: Waiting for error group managing segments... Dec 28 04:51:19.401967571 (541h22m19.273050616s) [noise=true , address=145.79.227.37:4566 , piece_id=7ZOB5524IYMURSQTQHXNHSAKAKWMAWM5MD5ZP75VGY3D2G7KJUAQ, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.419817650 (541h22m19.290900686s) [noise=true , address=145.79.227.24:4581 , piece_id=YWWZSG4JIFVXNZPOEGGBANFMC67JWIEB4G4ZCII4MBVKR2E4F46A, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.440252000 (541h22m19.311335035s) [noise=true , address=145.79.227.5:4221 , piece_id=VCO43DD4NWI3M2GVV57Z5QEENEAO5XBJDYVBRAVUZDQUNI244HUQ, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.481531402 (541h22m19.352614447s) [noise=true , address=145.79.227.52:4447 , piece_id=7XNVVY4PTG7DMEQQ7642SQSCFKYYLGB74CNQTRIVVFM26PMWVJUQ, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.485250945 (541h22m19.35633399s) [noise=true , address=145.79.227.27:4691 , piece_id=53DIZEGUEFJBCICGZNLXFQVVQBV2DAH2FEVYVR4CQLBLQRBYI5MA, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.494052805 (541h22m19.365135841s) [noise=true , address=145.79.227.45:4229 , piece_id=HNSVMW4TQPW3FQSPGJK3W4LAKHMCEVXPB73ANRF4BGTIPISZ7TZA, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.497031627 (541h22m19.368114672s) [noise=true , address=145.79.227.29:4941 , piece_id=QGCD3N3AWTIBLH7A4WLW6Q4QOZFPVGTVGEEPLUTWCTY3MBDPRUGA, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.655698854 (541h22m19.526781889s) [noise=true , address=77.90.31.10:4520 , piece_id=QI7L7F4JNKFB4HYIAGML42ZVDB6EYUHUSVMAKE6MJALWITNYOYVQ, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.683962011 (541h22m19.555045056s) [noise=true , address=145.79.227.16:4912 , piece_id=DW2G4Y7FJVSLQFWDFO27HPKUQBIYDW7NPRVCKWWKOW7JYHWAXCXQ, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:19.696741686 (541h22m19.567824721s) [noise=true , address=45.91.116.4:4940 , piece_id=J2FJL5OH2RRBOYALWDXMI47C6PMD7DOEFFRVWVRZW5OKROCLFDJA, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:20.369439464 (541h22m20.240522509s) [noise=true , address=45.91.116.14:4190 , piece_id=PWLQC4BAHZFGDGSNVDDYXL332UFTP5KWOOGVOA4NY67MXQSXONTQ, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:20.615307713 (541h22m20.486390758s) [noise=true , address=45.91.116.12:4228 , piece_id=7RXBJQEKBCS6AJ24II3Z4IXPAPXCYEJH7Q2CPWSM36A4WS4WQ4RA, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:20.790232397 (541h22m20.661315442s) [noise=true , address=45.91.116.10:4082 , piece_id=MDQ4WXQS2BH4X2QYWMXKWDTCINFD7DPBAIQ4ARPDHNFAKNBD4YRQ, seg_pos={0 0}, upload=474760]: Done uploading piece. err: <nil> Dec 28 04:51:20.790259298 (541h22m20.661342333s) [seg_pos={0 0}, upload=474760]: Segment reached optimal threshold of 13 pieces. Dec 28 04:51:20.790482583 (541h22m20.661565628s) [noise=true , address=45.91.116.13:4155 , piece_id=4BWX5W47CA5WMIXK2F5GXERNGGAHPTXGUMNRBUEKAVHPWMJTGX6A, seg_pos={0 0}, upload=474760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 04:51:20.790492963 (541h22m20.661576018s) [noise=true , address=45.91.116.13:4155 , piece_id=4BWX5W47CA5WMIXK2F5GXERNGGAHPTXGUMNRBUEKAVHPWMJTGX6A, seg_pos={0 0}, upload=474760]: Long tail cancellation due to optimal threshold Dec 28 04:51:20.790941006 (541h22m20.662024051s) [noise=true , address=77.90.31.18:4808 , piece_id=IFXJH2X4AB4ZL5IOTNIYDSNCNW3WORSQKG2NGTSNC62BOZCQZCAA, seg_pos={0 0}, upload=474760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 04:51:20.790951165 (541h22m20.66203421s) [noise=true , address=77.90.31.18:4808 , piece_id=IFXJH2X4AB4ZL5IOTNIYDSNCNW3WORSQKG2NGTSNC62BOZCQZCAA, seg_pos={0 0}, upload=474760]: Long tail cancellation due to optimal threshold Dec 28 04:51:20.791141208 (541h22m20.662224253s) [noise=true , address=45.91.116.5:4297 , piece_id=WAILBHZLOCEHDLYTBT765SXSDMUN7TQSCGSEXHTU4A3UJ4LV7XMA, seg_pos={0 0}, upload=474760]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 04:51:20.791150114 (541h22m20.662233159s) [noise=true , address=45.91.116.5:4297 , piece_id=WAILBHZLOCEHDLYTBT765SXSDMUN7TQSCGSEXHTU4A3UJ4LV7XMA, seg_pos={0 0}, upload=474760]: Long tail cancellation due to optimal threshold Dec 28 04:51:20.791173449 (541h22m20.662256494s) [seg_pos={0 0}, upload=474760]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:51:20.791257970 (541h22m20.662341015s) [upload=474760]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:51:20.791264091 (541h22m20.662347136s) [upload=474760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:51:21.089722300 (541h22m20.960805345s) [upload=474760]: Done uploading segments... Dec 28 04:51:21.089744112 (541h22m20.960827157s) [upload=474760]: Upload finished. err: <nil> Duration: 30.003520339s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....X] duration: 5.001512019s trace: https://jaeger.dev.storj.io/trace/52d304625cb819cc FAIL (2025-12-28 04:51:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00153383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X......X............................................] duration: 51.534909744s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:51:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 51.534948208s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....] duration: 4.742541517s trace: https://jaeger.dev.storj.io/trace/87dabe41ac7d744 FAIL (2025-12-28 04:50:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.742569189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..] duration: 6.916350664s trace: https://jaeger.dev.storj.io/trace/76d3cb71a870daf5 FAIL (2025-12-28 04:50:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA) context deadline exceeded ======LOGS====== Dec 28 04:48:20.407065073 (541h19m20.278148128s) [upload=474717]: Starting upload Dec 28 04:48:20.407153510 (541h19m20.278236545s) [upload=474717]: Done starting upload Dec 28 04:48:20.407208936 (541h19m20.278291981s) [upload=474717]: Uploading segments... Dec 28 04:48:20.407450787 (541h19m20.278533832s) [upload=474717]: Got next segment. Inline: false Dec 28 04:48:20.407497586 (541h19m20.278580621s) [upload=474717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:48:20.407504238 (541h19m20.278587273s) [upload=474717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:48:20.653666340 (541h19m20.524749385s) [seg_pos={0 0}, upload=474717]: Begin upload segment... Dec 28 04:48:20.653726986 (541h19m20.524810031s) [seg_pos={0 0}, upload=474717]: Done begin upload segment. Dec 28 04:48:20.653762343 (541h19m20.524845398s) [noise=true , address=145.79.227.42:4121 , piece_id=NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.653926025 (541h19m20.52500906s) [noise=true , address=145.79.227.19:4077 , piece_id=PM3V5WSRARWUULPLVOOYUWBRZR2DSYAFKYETRPGTFHXNM2AHXAVA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.654491902 (541h19m20.525574937s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.654781944 (541h19m20.525864979s) [noise=true , address=145.79.227.6:4258 , piece_id=5HFY43A5DXGZFBICCGRVY4WY4HIBTMUOR3T62GHKHWKFANMLLK7A, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655239065 (541h19m20.5263221s) [noise=true , address=145.79.227.12:4767 , piece_id=P6OINOSF7V24BMOEX3MOOI5EH3DJKWQ6QOD4UTJZAKV5ODJ2P7OQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655664004 (541h19m20.526747039s) [noise=true , address=45.91.116.7:4009 , piece_id=5ZUIOKFGDJJ3PSXP7RUOLTUEDUI74BNAJTBJFXLTB5MAK4PBIOHQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655673582 (541h19m20.526756647s) [noise=true , address=145.79.227.52:4446 , piece_id=63H5MFHMC46SVTPRF2ZOKRT4OO5IWMVI6STQ6UGPX47EF4FIA32A, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656025392 (541h19m20.527108437s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656056682 (541h19m20.527139727s) [noise=true , address=45.91.116.12:4225 , piece_id=2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656421237 (541h19m20.527504272s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656450041 (541h19m20.527533076s) [noise=true , address=45.91.116.13:4155 , piece_id=EGHAK3MGRVTZ2525LW5QLLZUQWJSKACSZGL6D4M7MFZI4IPRD2EA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656791571 (541h19m20.527874606s) [noise=true , address=145.79.227.14:4837 , piece_id=RTQX7O6XQJHONJ6P5KAIC25M7MHL5IWOXD2QMWOVNQGTJFJC6HWA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656863738 (541h19m20.527946773s) [noise=true , address=77.90.31.11:4555 , piece_id=JVETV3FSDHPQT2RMAWNQAV5AFDCWXXG6S7ZMHCD6MVLWDLTZ7IHA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657089288 (541h19m20.528172334s) [noise=true , address=145.79.227.41:4087 , piece_id=WBTY32UUIY7ZBNE5LSIGYPL6AO2LHIEUGTDJ4L6ZZFAY2R7242NQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657266546 (541h19m20.528349581s) [noise=true , address=45.91.116.9:4264 , piece_id=DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657414618 (541h19m20.528497653s) [noise=true , address=145.79.227.48:4303 , piece_id=RXPPB2RZW5NRDMF3F5AMFMRH5MLNOPEW3E6ZDMWCFBHEKL722XHQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:22.371408405 (541h19m22.24249144s) [upload=474717]: Next returned nil segment Dec 28 04:48:22.371476605 (541h19m22.24255964s) [upload=474717]: Waiting for error group managing segments... Dec 28 04:48:22.651847617 (541h19m22.522930662s) [noise=true , address=145.79.227.6:4258 , piece_id=5HFY43A5DXGZFBICCGRVY4WY4HIBTMUOR3T62GHKHWKFANMLLK7A, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.662179490 (541h19m22.533262535s) [noise=true , address=145.79.227.19:4077 , piece_id=PM3V5WSRARWUULPLVOOYUWBRZR2DSYAFKYETRPGTFHXNM2AHXAVA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.670461019 (541h19m22.541544064s) [noise=true , address=145.79.227.48:4303 , piece_id=RXPPB2RZW5NRDMF3F5AMFMRH5MLNOPEW3E6ZDMWCFBHEKL722XHQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.671464751 (541h19m22.542547796s) [noise=true , address=145.79.227.52:4446 , piece_id=63H5MFHMC46SVTPRF2ZOKRT4OO5IWMVI6STQ6UGPX47EF4FIA32A, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.679855217 (541h19m22.550938263s) [noise=true , address=145.79.227.42:4121 , piece_id=NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.683564471 (541h19m22.554647516s) [noise=true , address=145.79.227.12:4767 , piece_id=P6OINOSF7V24BMOEX3MOOI5EH3DJKWQ6QOD4UTJZAKV5ODJ2P7OQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.689103687 (541h19m22.560186723s) [noise=true , address=145.79.227.14:4837 , piece_id=RTQX7O6XQJHONJ6P5KAIC25M7MHL5IWOXD2QMWOVNQGTJFJC6HWA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.693514627 (541h19m22.564597672s) [noise=true , address=145.79.227.41:4087 , piece_id=WBTY32UUIY7ZBNE5LSIGYPL6AO2LHIEUGTDJ4L6ZZFAY2R7242NQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.888240856 (541h19m22.759323902s) [noise=true , address=45.91.116.12:4225 , piece_id=2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.944636674 (541h19m22.815719709s) [noise=true , address=45.91.116.9:4264 , piece_id=DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.021647054 (541h19m22.892730089s) [noise=true , address=45.91.116.13:4155 , piece_id=EGHAK3MGRVTZ2525LW5QLLZUQWJSKACSZGL6D4M7MFZI4IPRD2EA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.036052845 (541h19m22.90713589s) [noise=true , address=45.91.116.7:4009 , piece_id=5ZUIOKFGDJJ3PSXP7RUOLTUEDUI74BNAJTBJFXLTB5MAK4PBIOHQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.071793601 (541h19m22.942876646s) [noise=true , address=77.90.31.11:4555 , piece_id=JVETV3FSDHPQT2RMAWNQAV5AFDCWXXG6S7ZMHCD6MVLWDLTZ7IHA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.071821724 (541h19m22.94290477s) [seg_pos={0 0}, upload=474717]: Segment reached optimal threshold of 13 pieces. Dec 28 04:48:23.072060169 (541h19m22.943143204s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072071480 (541h19m22.943154515s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072174757 (541h19m22.943257792s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072196398 (541h19m22.943279433s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072328138 (541h19m22.943411173s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072334742 (541h19m22.943417787s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072351002 (541h19m22.943434037s) [seg_pos={0 0}, upload=474717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:48:23.072425864 (541h19m22.9435089s) [upload=474717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:48:23.072431074 (541h19m22.943514109s) [upload=474717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:48:23.369827395 (541h19m23.24091044s) [upload=474717]: Done uploading segments... Dec 28 04:48:23.369848354 (541h19m23.240931399s) [upload=474717]: Upload finished. err: <nil> Duration: 6.916377386s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX....] duration: 3.666631572s trace: https://jaeger.dev.storj.io/trace/87dabe41ac7d744 FAIL (2025-12-28 04:49:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.66665177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X.] duration: 6.155824754s trace: https://jaeger.dev.storj.io/trace/76d3cb71a870daf5 FAIL (2025-12-28 04:49:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA) context deadline exceeded ======LOGS====== Dec 28 04:48:20.407065073 (541h19m20.278148128s) [upload=474717]: Starting upload Dec 28 04:48:20.407153510 (541h19m20.278236545s) [upload=474717]: Done starting upload Dec 28 04:48:20.407208936 (541h19m20.278291981s) [upload=474717]: Uploading segments... Dec 28 04:48:20.407450787 (541h19m20.278533832s) [upload=474717]: Got next segment. Inline: false Dec 28 04:48:20.407497586 (541h19m20.278580621s) [upload=474717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:48:20.407504238 (541h19m20.278587273s) [upload=474717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:48:20.653666340 (541h19m20.524749385s) [seg_pos={0 0}, upload=474717]: Begin upload segment... Dec 28 04:48:20.653726986 (541h19m20.524810031s) [seg_pos={0 0}, upload=474717]: Done begin upload segment. Dec 28 04:48:20.653762343 (541h19m20.524845398s) [noise=true , address=145.79.227.42:4121 , piece_id=NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.653926025 (541h19m20.52500906s) [noise=true , address=145.79.227.19:4077 , piece_id=PM3V5WSRARWUULPLVOOYUWBRZR2DSYAFKYETRPGTFHXNM2AHXAVA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.654491902 (541h19m20.525574937s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.654781944 (541h19m20.525864979s) [noise=true , address=145.79.227.6:4258 , piece_id=5HFY43A5DXGZFBICCGRVY4WY4HIBTMUOR3T62GHKHWKFANMLLK7A, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655239065 (541h19m20.5263221s) [noise=true , address=145.79.227.12:4767 , piece_id=P6OINOSF7V24BMOEX3MOOI5EH3DJKWQ6QOD4UTJZAKV5ODJ2P7OQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655664004 (541h19m20.526747039s) [noise=true , address=45.91.116.7:4009 , piece_id=5ZUIOKFGDJJ3PSXP7RUOLTUEDUI74BNAJTBJFXLTB5MAK4PBIOHQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655673582 (541h19m20.526756647s) [noise=true , address=145.79.227.52:4446 , piece_id=63H5MFHMC46SVTPRF2ZOKRT4OO5IWMVI6STQ6UGPX47EF4FIA32A, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656025392 (541h19m20.527108437s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656056682 (541h19m20.527139727s) [noise=true , address=45.91.116.12:4225 , piece_id=2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656421237 (541h19m20.527504272s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656450041 (541h19m20.527533076s) [noise=true , address=45.91.116.13:4155 , piece_id=EGHAK3MGRVTZ2525LW5QLLZUQWJSKACSZGL6D4M7MFZI4IPRD2EA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656791571 (541h19m20.527874606s) [noise=true , address=145.79.227.14:4837 , piece_id=RTQX7O6XQJHONJ6P5KAIC25M7MHL5IWOXD2QMWOVNQGTJFJC6HWA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656863738 (541h19m20.527946773s) [noise=true , address=77.90.31.11:4555 , piece_id=JVETV3FSDHPQT2RMAWNQAV5AFDCWXXG6S7ZMHCD6MVLWDLTZ7IHA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657089288 (541h19m20.528172334s) [noise=true , address=145.79.227.41:4087 , piece_id=WBTY32UUIY7ZBNE5LSIGYPL6AO2LHIEUGTDJ4L6ZZFAY2R7242NQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657266546 (541h19m20.528349581s) [noise=true , address=45.91.116.9:4264 , piece_id=DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657414618 (541h19m20.528497653s) [noise=true , address=145.79.227.48:4303 , piece_id=RXPPB2RZW5NRDMF3F5AMFMRH5MLNOPEW3E6ZDMWCFBHEKL722XHQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:22.371408405 (541h19m22.24249144s) [upload=474717]: Next returned nil segment Dec 28 04:48:22.371476605 (541h19m22.24255964s) [upload=474717]: Waiting for error group managing segments... Dec 28 04:48:22.651847617 (541h19m22.522930662s) [noise=true , address=145.79.227.6:4258 , piece_id=5HFY43A5DXGZFBICCGRVY4WY4HIBTMUOR3T62GHKHWKFANMLLK7A, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.662179490 (541h19m22.533262535s) [noise=true , address=145.79.227.19:4077 , piece_id=PM3V5WSRARWUULPLVOOYUWBRZR2DSYAFKYETRPGTFHXNM2AHXAVA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.670461019 (541h19m22.541544064s) [noise=true , address=145.79.227.48:4303 , piece_id=RXPPB2RZW5NRDMF3F5AMFMRH5MLNOPEW3E6ZDMWCFBHEKL722XHQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.671464751 (541h19m22.542547796s) [noise=true , address=145.79.227.52:4446 , piece_id=63H5MFHMC46SVTPRF2ZOKRT4OO5IWMVI6STQ6UGPX47EF4FIA32A, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.679855217 (541h19m22.550938263s) [noise=true , address=145.79.227.42:4121 , piece_id=NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.683564471 (541h19m22.554647516s) [noise=true , address=145.79.227.12:4767 , piece_id=P6OINOSF7V24BMOEX3MOOI5EH3DJKWQ6QOD4UTJZAKV5ODJ2P7OQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.689103687 (541h19m22.560186723s) [noise=true , address=145.79.227.14:4837 , piece_id=RTQX7O6XQJHONJ6P5KAIC25M7MHL5IWOXD2QMWOVNQGTJFJC6HWA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.693514627 (541h19m22.564597672s) [noise=true , address=145.79.227.41:4087 , piece_id=WBTY32UUIY7ZBNE5LSIGYPL6AO2LHIEUGTDJ4L6ZZFAY2R7242NQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.888240856 (541h19m22.759323902s) [noise=true , address=45.91.116.12:4225 , piece_id=2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.944636674 (541h19m22.815719709s) [noise=true , address=45.91.116.9:4264 , piece_id=DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.021647054 (541h19m22.892730089s) [noise=true , address=45.91.116.13:4155 , piece_id=EGHAK3MGRVTZ2525LW5QLLZUQWJSKACSZGL6D4M7MFZI4IPRD2EA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.036052845 (541h19m22.90713589s) [noise=true , address=45.91.116.7:4009 , piece_id=5ZUIOKFGDJJ3PSXP7RUOLTUEDUI74BNAJTBJFXLTB5MAK4PBIOHQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.071793601 (541h19m22.942876646s) [noise=true , address=77.90.31.11:4555 , piece_id=JVETV3FSDHPQT2RMAWNQAV5AFDCWXXG6S7ZMHCD6MVLWDLTZ7IHA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.071821724 (541h19m22.94290477s) [seg_pos={0 0}, upload=474717]: Segment reached optimal threshold of 13 pieces. Dec 28 04:48:23.072060169 (541h19m22.943143204s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072071480 (541h19m22.943154515s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072174757 (541h19m22.943257792s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072196398 (541h19m22.943279433s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072328138 (541h19m22.943411173s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072334742 (541h19m22.943417787s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072351002 (541h19m22.943434037s) [seg_pos={0 0}, upload=474717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:48:23.072425864 (541h19m22.9435089s) [upload=474717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:48:23.072431074 (541h19m22.943514109s) [upload=474717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:48:23.369827395 (541h19m23.24091044s) [upload=474717]: Done uploading segments... Dec 28 04:48:23.369848354 (541h19m23.240931399s) [upload=474717]: Upload finished. err: <nil> Duration: 6.155849883s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX...] duration: 3.55437144s trace: https://jaeger.dev.storj.io/trace/87dabe41ac7d744 FAIL (2025-12-28 04:49:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554431665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X......X...........................................] duration: 34.270184232s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:49:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 34.27020447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X...X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX..] duration: 3.632807094s trace: https://jaeger.dev.storj.io/trace/87dabe41ac7d744 FAIL (2025-12-28 04:48:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.632828214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X......X..........................................] duration: 34.418662542s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:48:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 34.418689373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X] duration: 30.004371828s trace: https://jaeger.dev.storj.io/trace/76d3cb71a870daf5 FAIL (2025-12-28 04:48:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA) context deadline exceeded ======LOGS====== Dec 28 04:48:20.407065073 (541h19m20.278148128s) [upload=474717]: Starting upload Dec 28 04:48:20.407153510 (541h19m20.278236545s) [upload=474717]: Done starting upload Dec 28 04:48:20.407208936 (541h19m20.278291981s) [upload=474717]: Uploading segments... Dec 28 04:48:20.407450787 (541h19m20.278533832s) [upload=474717]: Got next segment. Inline: false Dec 28 04:48:20.407497586 (541h19m20.278580621s) [upload=474717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:48:20.407504238 (541h19m20.278587273s) [upload=474717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:48:20.653666340 (541h19m20.524749385s) [seg_pos={0 0}, upload=474717]: Begin upload segment... Dec 28 04:48:20.653726986 (541h19m20.524810031s) [seg_pos={0 0}, upload=474717]: Done begin upload segment. Dec 28 04:48:20.653762343 (541h19m20.524845398s) [noise=true , address=145.79.227.42:4121 , piece_id=NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.653926025 (541h19m20.52500906s) [noise=true , address=145.79.227.19:4077 , piece_id=PM3V5WSRARWUULPLVOOYUWBRZR2DSYAFKYETRPGTFHXNM2AHXAVA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.654491902 (541h19m20.525574937s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.654781944 (541h19m20.525864979s) [noise=true , address=145.79.227.6:4258 , piece_id=5HFY43A5DXGZFBICCGRVY4WY4HIBTMUOR3T62GHKHWKFANMLLK7A, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655239065 (541h19m20.5263221s) [noise=true , address=145.79.227.12:4767 , piece_id=P6OINOSF7V24BMOEX3MOOI5EH3DJKWQ6QOD4UTJZAKV5ODJ2P7OQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655664004 (541h19m20.526747039s) [noise=true , address=45.91.116.7:4009 , piece_id=5ZUIOKFGDJJ3PSXP7RUOLTUEDUI74BNAJTBJFXLTB5MAK4PBIOHQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.655673582 (541h19m20.526756647s) [noise=true , address=145.79.227.52:4446 , piece_id=63H5MFHMC46SVTPRF2ZOKRT4OO5IWMVI6STQ6UGPX47EF4FIA32A, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656025392 (541h19m20.527108437s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656056682 (541h19m20.527139727s) [noise=true , address=45.91.116.12:4225 , piece_id=2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656421237 (541h19m20.527504272s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656450041 (541h19m20.527533076s) [noise=true , address=45.91.116.13:4155 , piece_id=EGHAK3MGRVTZ2525LW5QLLZUQWJSKACSZGL6D4M7MFZI4IPRD2EA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656791571 (541h19m20.527874606s) [noise=true , address=145.79.227.14:4837 , piece_id=RTQX7O6XQJHONJ6P5KAIC25M7MHL5IWOXD2QMWOVNQGTJFJC6HWA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.656863738 (541h19m20.527946773s) [noise=true , address=77.90.31.11:4555 , piece_id=JVETV3FSDHPQT2RMAWNQAV5AFDCWXXG6S7ZMHCD6MVLWDLTZ7IHA, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657089288 (541h19m20.528172334s) [noise=true , address=145.79.227.41:4087 , piece_id=WBTY32UUIY7ZBNE5LSIGYPL6AO2LHIEUGTDJ4L6ZZFAY2R7242NQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657266546 (541h19m20.528349581s) [noise=true , address=45.91.116.9:4264 , piece_id=DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:20.657414618 (541h19m20.528497653s) [noise=true , address=145.79.227.48:4303 , piece_id=RXPPB2RZW5NRDMF3F5AMFMRH5MLNOPEW3E6ZDMWCFBHEKL722XHQ, seg_pos={0 0}, upload=474717]: Uploading piece... Dec 28 04:48:22.371408405 (541h19m22.24249144s) [upload=474717]: Next returned nil segment Dec 28 04:48:22.371476605 (541h19m22.24255964s) [upload=474717]: Waiting for error group managing segments... Dec 28 04:48:22.651847617 (541h19m22.522930662s) [noise=true , address=145.79.227.6:4258 , piece_id=5HFY43A5DXGZFBICCGRVY4WY4HIBTMUOR3T62GHKHWKFANMLLK7A, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.662179490 (541h19m22.533262535s) [noise=true , address=145.79.227.19:4077 , piece_id=PM3V5WSRARWUULPLVOOYUWBRZR2DSYAFKYETRPGTFHXNM2AHXAVA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.670461019 (541h19m22.541544064s) [noise=true , address=145.79.227.48:4303 , piece_id=RXPPB2RZW5NRDMF3F5AMFMRH5MLNOPEW3E6ZDMWCFBHEKL722XHQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.671464751 (541h19m22.542547796s) [noise=true , address=145.79.227.52:4446 , piece_id=63H5MFHMC46SVTPRF2ZOKRT4OO5IWMVI6STQ6UGPX47EF4FIA32A, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.679855217 (541h19m22.550938263s) [noise=true , address=145.79.227.42:4121 , piece_id=NLPFWDNCOJFH56HGQXBAFSZ5HIPEE2JZCYTKZ3XDWRDXNQIM5OQA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.683564471 (541h19m22.554647516s) [noise=true , address=145.79.227.12:4767 , piece_id=P6OINOSF7V24BMOEX3MOOI5EH3DJKWQ6QOD4UTJZAKV5ODJ2P7OQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.689103687 (541h19m22.560186723s) [noise=true , address=145.79.227.14:4837 , piece_id=RTQX7O6XQJHONJ6P5KAIC25M7MHL5IWOXD2QMWOVNQGTJFJC6HWA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.693514627 (541h19m22.564597672s) [noise=true , address=145.79.227.41:4087 , piece_id=WBTY32UUIY7ZBNE5LSIGYPL6AO2LHIEUGTDJ4L6ZZFAY2R7242NQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.888240856 (541h19m22.759323902s) [noise=true , address=45.91.116.12:4225 , piece_id=2G7VXN77ZEPUVNOU27CVL272QU263TKLFEIYSQMWIHAO3DQRNPLQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:22.944636674 (541h19m22.815719709s) [noise=true , address=45.91.116.9:4264 , piece_id=DRALDAUILS2EEHAWUCSYEKBPDI5U3V6NTOILOV5J437KK65BU5RQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.021647054 (541h19m22.892730089s) [noise=true , address=45.91.116.13:4155 , piece_id=EGHAK3MGRVTZ2525LW5QLLZUQWJSKACSZGL6D4M7MFZI4IPRD2EA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.036052845 (541h19m22.90713589s) [noise=true , address=45.91.116.7:4009 , piece_id=5ZUIOKFGDJJ3PSXP7RUOLTUEDUI74BNAJTBJFXLTB5MAK4PBIOHQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.071793601 (541h19m22.942876646s) [noise=true , address=77.90.31.11:4555 , piece_id=JVETV3FSDHPQT2RMAWNQAV5AFDCWXXG6S7ZMHCD6MVLWDLTZ7IHA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: <nil> Dec 28 04:48:23.071821724 (541h19m22.94290477s) [seg_pos={0 0}, upload=474717]: Segment reached optimal threshold of 13 pieces. Dec 28 04:48:23.072060169 (541h19m22.943143204s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072071480 (541h19m22.943154515s) [noise=true , address=145.79.227.43:4159 , piece_id=S5D3GHKWPGVMVWZGJ2AL73OBI2X2V4SH3T7AX4TXZOOAUBYQRJKQ, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072174757 (541h19m22.943257792s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072196398 (541h19m22.943279433s) [noise=true , address=145.79.227.50:4375 , piece_id=SAG3SSCLGPXSNBWFHSKR5FL7RYARSOOERXAZ2ORPWKM7JE3XQXZA, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072328138 (541h19m22.943411173s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 28 04:48:23.072334742 (541h19m22.943417787s) [noise=true , address=77.90.31.10:4520 , piece_id=PIP6AZ2XE4HWV7M3XQEUPM4CNJCT2OJNRL3B23764MGA5C422Z5Q, seg_pos={0 0}, upload=474717]: Long tail cancellation due to optimal threshold Dec 28 04:48:23.072351002 (541h19m22.943434037s) [seg_pos={0 0}, upload=474717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:48:23.072425864 (541h19m22.9435089s) [upload=474717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:48:23.072431074 (541h19m22.943514109s) [upload=474717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:48:23.369827395 (541h19m23.24091044s) [upload=474717]: Done uploading segments... Dec 28 04:48:23.369848354 (541h19m23.240931399s) [upload=474717]: Upload finished. err: <nil> Duration: 30.004397536s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX...X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.] duration: 3.780363096s trace: https://jaeger.dev.storj.io/trace/87dabe41ac7d744 FAIL (2025-12-28 04:48:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.780413812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X......X.........................................] duration: 42.581590742s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:48:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 42.581620348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...] duration: 6.313995354s trace: https://jaeger.dev.storj.io/trace/78399034b5a785e0 FAIL (2025-12-28 04:47:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:45:44.734142397 (541h16m44.605225442s) [upload=474680]: Starting upload Dec 28 04:45:44.734224875 (541h16m44.60530792s) [upload=474680]: Done starting upload Dec 28 04:45:44.734367316 (541h16m44.605450351s) [upload=474680]: Uploading segments... Dec 28 04:45:44.734508254 (541h16m44.605591289s) [upload=474680]: Got next segment. Inline: false Dec 28 04:45:44.734551076 (541h16m44.605634111s) [upload=474680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:45:44.734570583 (541h16m44.605653618s) [upload=474680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:45:44.977050414 (541h16m44.848133459s) [seg_pos={0 0}, upload=474680]: Begin upload segment... Dec 28 04:45:44.977106280 (541h16m44.848189315s) [seg_pos={0 0}, upload=474680]: Done begin upload segment. Dec 28 04:45:44.977121670 (541h16m44.848204715s) [noise=true , address=45.91.116.8:4048 , piece_id=NJFEVWXIOBZ2EXRY2DKNMXB6JZRPVCPLS4S2NYUO4KGRL2CLPNYA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.977137048 (541h16m44.848220094s) [noise=true , address=145.79.227.15:4873 , piece_id=RKG5IGVNP6KHBTTVYVDSSWREBSFDNCMHFJ44CCQMVGJZKQLMAWXQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.977338673 (541h16m44.848421728s) [noise=true , address=45.91.116.6:4973 , piece_id=UGOYUOT3L2YU3WFGSZ3WP2JALZLLSF4MG5VCSIIZIQAQ3XPMAQ6A, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978028276 (541h16m44.849111321s) [noise=true , address=77.90.31.19:4844 , piece_id=IEFP4SXS44ER75RSZQEKDHFQ2L64LICEJVRWZXVUEGFZLZBENHHA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978423648 (541h16m44.849506693s) [noise=true , address=145.79.227.34:4905 , piece_id=FD6K5B32OKFWBROKPBPSJZ63R5NMGSIIHVD3RN5BSTK7V4TUTSFA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978878925 (541h16m44.84996197s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.979289828 (541h16m44.850372873s) [noise=true , address=77.90.31.21:4880 , piece_id=UMUGQUFOGTJ4YJPDKWUUJYMCOTUTH2DNDAMX2R4LN4GRVBHF4FCQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.979687725 (541h16m44.85077077s) [noise=true , address=77.90.31.13:4629 , piece_id=D55COMNZTRSQQGE7CHTMBX7DTWRZX3URAVSBGVFJAHNFZO6V4MKQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.980122744 (541h16m44.851205789s) [noise=true , address=45.91.116.11:4119 , piece_id=425WPUPDWNL3ACQ47D7LP2IJXVMN6TKWCXKHENIKX5W77WT7Q3BQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.981913293 (541h16m44.852996338s) [noise=true , address=145.79.227.51:4409 , piece_id=MWQLWVVRJMGWBMDGPY6ZKUD3GEGCI3GLA4L5NC3XQ6PCJSNGCTQA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.984102331 (541h16m44.855185367s) [noise=true , address=145.79.227.36:4518 , piece_id=YDCVMGJ4H54O44IPWXCNKMHS6GINDDF5M7TWZEBED53IUZEPV4YQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.986049640 (541h16m44.857132675s) [noise=true , address=145.79.227.47:4267 , piece_id=QCBZRWP2T7OR3Y2XV4FXGQKEYFTIFGS6CMAGBBGABKXMXAPTBM2A, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.987714420 (541h16m44.858797465s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.989278367 (541h16m44.860361412s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.990880558 (541h16m44.861963603s) [noise=true , address=145.79.227.10:4403 , piece_id=S2GXANYUM4MQX4LNAKUTH57MZZ4YN4Y3JJFOURJXPIVDKLNHHTHQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.992595845 (541h16m44.86367888s) [noise=true , address=145.79.227.3:4151 , piece_id=WB6DOLKIPKVIMX7J7JS72VEN5O2YNDH6H7GUFUWF3CJFIGRAMYWQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:46.704832736 (541h16m46.575915771s) [upload=474680]: Next returned nil segment Dec 28 04:45:46.704853466 (541h16m46.575936511s) [upload=474680]: Waiting for error group managing segments... Dec 28 04:45:46.983137443 (541h16m46.854220478s) [noise=true , address=145.79.227.15:4873 , piece_id=RKG5IGVNP6KHBTTVYVDSSWREBSFDNCMHFJ44CCQMVGJZKQLMAWXQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:46.996830186 (541h16m46.867913231s) [noise=true , address=145.79.227.36:4518 , piece_id=YDCVMGJ4H54O44IPWXCNKMHS6GINDDF5M7TWZEBED53IUZEPV4YQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:46.997882530 (541h16m46.868965575s) [noise=true , address=145.79.227.10:4403 , piece_id=S2GXANYUM4MQX4LNAKUTH57MZZ4YN4Y3JJFOURJXPIVDKLNHHTHQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.008211007 (541h16m46.879294052s) [noise=true , address=145.79.227.47:4267 , piece_id=QCBZRWP2T7OR3Y2XV4FXGQKEYFTIFGS6CMAGBBGABKXMXAPTBM2A, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.039492062 (541h16m46.910575097s) [noise=true , address=145.79.227.3:4151 , piece_id=WB6DOLKIPKVIMX7J7JS72VEN5O2YNDH6H7GUFUWF3CJFIGRAMYWQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.055717778 (541h16m46.926800823s) [noise=true , address=145.79.227.51:4409 , piece_id=MWQLWVVRJMGWBMDGPY6ZKUD3GEGCI3GLA4L5NC3XQ6PCJSNGCTQA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.267464488 (541h16m47.138547553s) [noise=true , address=77.90.31.13:4629 , piece_id=D55COMNZTRSQQGE7CHTMBX7DTWRZX3URAVSBGVFJAHNFZO6V4MKQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.273368700 (541h16m47.144451735s) [noise=true , address=45.91.116.6:4973 , piece_id=UGOYUOT3L2YU3WFGSZ3WP2JALZLLSF4MG5VCSIIZIQAQ3XPMAQ6A, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.279613912 (541h16m47.150696947s) [noise=true , address=45.91.116.11:4119 , piece_id=425WPUPDWNL3ACQ47D7LP2IJXVMN6TKWCXKHENIKX5W77WT7Q3BQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.400787305 (541h16m47.27187035s) [noise=true , address=45.91.116.8:4048 , piece_id=NJFEVWXIOBZ2EXRY2DKNMXB6JZRPVCPLS4S2NYUO4KGRL2CLPNYA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.259027010 (541h16m48.130110055s) [noise=true , address=145.79.227.34:4905 , piece_id=FD6K5B32OKFWBROKPBPSJZ63R5NMGSIIHVD3RN5BSTK7V4TUTSFA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.783117661 (541h16m48.654200696s) [noise=true , address=77.90.31.21:4880 , piece_id=UMUGQUFOGTJ4YJPDKWUUJYMCOTUTH2DNDAMX2R4LN4GRVBHF4FCQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.885156761 (541h16m48.756239796s) [noise=true , address=77.90.31.19:4844 , piece_id=IEFP4SXS44ER75RSZQEKDHFQ2L64LICEJVRWZXVUEGFZLZBENHHA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.885178291 (541h16m48.756261326s) [seg_pos={0 0}, upload=474680]: Segment reached optimal threshold of 13 pieces. Dec 28 04:45:48.885435412 (541h16m48.756518447s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44946->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:45:48.885442094 (541h16m48.756525139s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885512779 (541h16m48.756595814s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 04:45:48.885535542 (541h16m48.756618577s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885666361 (541h16m48.756749396s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:45:48.885672783 (541h16m48.756755818s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885687962 (541h16m48.756770997s) [seg_pos={0 0}, upload=474680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:45:48.885786730 (541h16m48.756869766s) [upload=474680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:45:48.885791449 (541h16m48.756874494s) [upload=474680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:45:49.175373480 (541h16m49.046456515s) [upload=474680]: Done uploading segments... Dec 28 04:45:49.175395492 (541h16m49.046478527s) [upload=474680]: Upload finished. err: <nil> Duration: 6.314022737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XX...X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX] duration: 5.000732272s trace: https://jaeger.dev.storj.io/trace/87dabe41ac7d744 FAIL (2025-12-28 04:47:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00075773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X......X........................................] duration: 41.862306013s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:47:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 41.862339226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XXXXXXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X..] duration: 6.021984851s trace: https://jaeger.dev.storj.io/trace/78399034b5a785e0 FAIL (2025-12-28 04:46:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:45:44.734142397 (541h16m44.605225442s) [upload=474680]: Starting upload Dec 28 04:45:44.734224875 (541h16m44.60530792s) [upload=474680]: Done starting upload Dec 28 04:45:44.734367316 (541h16m44.605450351s) [upload=474680]: Uploading segments... Dec 28 04:45:44.734508254 (541h16m44.605591289s) [upload=474680]: Got next segment. Inline: false Dec 28 04:45:44.734551076 (541h16m44.605634111s) [upload=474680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:45:44.734570583 (541h16m44.605653618s) [upload=474680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:45:44.977050414 (541h16m44.848133459s) [seg_pos={0 0}, upload=474680]: Begin upload segment... Dec 28 04:45:44.977106280 (541h16m44.848189315s) [seg_pos={0 0}, upload=474680]: Done begin upload segment. Dec 28 04:45:44.977121670 (541h16m44.848204715s) [noise=true , address=45.91.116.8:4048 , piece_id=NJFEVWXIOBZ2EXRY2DKNMXB6JZRPVCPLS4S2NYUO4KGRL2CLPNYA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.977137048 (541h16m44.848220094s) [noise=true , address=145.79.227.15:4873 , piece_id=RKG5IGVNP6KHBTTVYVDSSWREBSFDNCMHFJ44CCQMVGJZKQLMAWXQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.977338673 (541h16m44.848421728s) [noise=true , address=45.91.116.6:4973 , piece_id=UGOYUOT3L2YU3WFGSZ3WP2JALZLLSF4MG5VCSIIZIQAQ3XPMAQ6A, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978028276 (541h16m44.849111321s) [noise=true , address=77.90.31.19:4844 , piece_id=IEFP4SXS44ER75RSZQEKDHFQ2L64LICEJVRWZXVUEGFZLZBENHHA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978423648 (541h16m44.849506693s) [noise=true , address=145.79.227.34:4905 , piece_id=FD6K5B32OKFWBROKPBPSJZ63R5NMGSIIHVD3RN5BSTK7V4TUTSFA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978878925 (541h16m44.84996197s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.979289828 (541h16m44.850372873s) [noise=true , address=77.90.31.21:4880 , piece_id=UMUGQUFOGTJ4YJPDKWUUJYMCOTUTH2DNDAMX2R4LN4GRVBHF4FCQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.979687725 (541h16m44.85077077s) [noise=true , address=77.90.31.13:4629 , piece_id=D55COMNZTRSQQGE7CHTMBX7DTWRZX3URAVSBGVFJAHNFZO6V4MKQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.980122744 (541h16m44.851205789s) [noise=true , address=45.91.116.11:4119 , piece_id=425WPUPDWNL3ACQ47D7LP2IJXVMN6TKWCXKHENIKX5W77WT7Q3BQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.981913293 (541h16m44.852996338s) [noise=true , address=145.79.227.51:4409 , piece_id=MWQLWVVRJMGWBMDGPY6ZKUD3GEGCI3GLA4L5NC3XQ6PCJSNGCTQA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.984102331 (541h16m44.855185367s) [noise=true , address=145.79.227.36:4518 , piece_id=YDCVMGJ4H54O44IPWXCNKMHS6GINDDF5M7TWZEBED53IUZEPV4YQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.986049640 (541h16m44.857132675s) [noise=true , address=145.79.227.47:4267 , piece_id=QCBZRWP2T7OR3Y2XV4FXGQKEYFTIFGS6CMAGBBGABKXMXAPTBM2A, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.987714420 (541h16m44.858797465s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.989278367 (541h16m44.860361412s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.990880558 (541h16m44.861963603s) [noise=true , address=145.79.227.10:4403 , piece_id=S2GXANYUM4MQX4LNAKUTH57MZZ4YN4Y3JJFOURJXPIVDKLNHHTHQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.992595845 (541h16m44.86367888s) [noise=true , address=145.79.227.3:4151 , piece_id=WB6DOLKIPKVIMX7J7JS72VEN5O2YNDH6H7GUFUWF3CJFIGRAMYWQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:46.704832736 (541h16m46.575915771s) [upload=474680]: Next returned nil segment Dec 28 04:45:46.704853466 (541h16m46.575936511s) [upload=474680]: Waiting for error group managing segments... Dec 28 04:45:46.983137443 (541h16m46.854220478s) [noise=true , address=145.79.227.15:4873 , piece_id=RKG5IGVNP6KHBTTVYVDSSWREBSFDNCMHFJ44CCQMVGJZKQLMAWXQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:46.996830186 (541h16m46.867913231s) [noise=true , address=145.79.227.36:4518 , piece_id=YDCVMGJ4H54O44IPWXCNKMHS6GINDDF5M7TWZEBED53IUZEPV4YQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:46.997882530 (541h16m46.868965575s) [noise=true , address=145.79.227.10:4403 , piece_id=S2GXANYUM4MQX4LNAKUTH57MZZ4YN4Y3JJFOURJXPIVDKLNHHTHQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.008211007 (541h16m46.879294052s) [noise=true , address=145.79.227.47:4267 , piece_id=QCBZRWP2T7OR3Y2XV4FXGQKEYFTIFGS6CMAGBBGABKXMXAPTBM2A, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.039492062 (541h16m46.910575097s) [noise=true , address=145.79.227.3:4151 , piece_id=WB6DOLKIPKVIMX7J7JS72VEN5O2YNDH6H7GUFUWF3CJFIGRAMYWQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.055717778 (541h16m46.926800823s) [noise=true , address=145.79.227.51:4409 , piece_id=MWQLWVVRJMGWBMDGPY6ZKUD3GEGCI3GLA4L5NC3XQ6PCJSNGCTQA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.267464488 (541h16m47.138547553s) [noise=true , address=77.90.31.13:4629 , piece_id=D55COMNZTRSQQGE7CHTMBX7DTWRZX3URAVSBGVFJAHNFZO6V4MKQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.273368700 (541h16m47.144451735s) [noise=true , address=45.91.116.6:4973 , piece_id=UGOYUOT3L2YU3WFGSZ3WP2JALZLLSF4MG5VCSIIZIQAQ3XPMAQ6A, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.279613912 (541h16m47.150696947s) [noise=true , address=45.91.116.11:4119 , piece_id=425WPUPDWNL3ACQ47D7LP2IJXVMN6TKWCXKHENIKX5W77WT7Q3BQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.400787305 (541h16m47.27187035s) [noise=true , address=45.91.116.8:4048 , piece_id=NJFEVWXIOBZ2EXRY2DKNMXB6JZRPVCPLS4S2NYUO4KGRL2CLPNYA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.259027010 (541h16m48.130110055s) [noise=true , address=145.79.227.34:4905 , piece_id=FD6K5B32OKFWBROKPBPSJZ63R5NMGSIIHVD3RN5BSTK7V4TUTSFA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.783117661 (541h16m48.654200696s) [noise=true , address=77.90.31.21:4880 , piece_id=UMUGQUFOGTJ4YJPDKWUUJYMCOTUTH2DNDAMX2R4LN4GRVBHF4FCQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.885156761 (541h16m48.756239796s) [noise=true , address=77.90.31.19:4844 , piece_id=IEFP4SXS44ER75RSZQEKDHFQ2L64LICEJVRWZXVUEGFZLZBENHHA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.885178291 (541h16m48.756261326s) [seg_pos={0 0}, upload=474680]: Segment reached optimal threshold of 13 pieces. Dec 28 04:45:48.885435412 (541h16m48.756518447s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44946->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:45:48.885442094 (541h16m48.756525139s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885512779 (541h16m48.756595814s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 04:45:48.885535542 (541h16m48.756618577s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885666361 (541h16m48.756749396s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:45:48.885672783 (541h16m48.756755818s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885687962 (541h16m48.756770997s) [seg_pos={0 0}, upload=474680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:45:48.885786730 (541h16m48.756869766s) [upload=474680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:45:48.885791449 (541h16m48.756874494s) [upload=474680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:45:49.175373480 (541h16m49.046456515s) [upload=474680]: Done uploading segments... Dec 28 04:45:49.175395492 (541h16m49.046478527s) [upload=474680]: Upload finished. err: <nil> Duration: 6.022011872s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..XX...X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXX] duration: 5.000926143s trace: https://jaeger.dev.storj.io/trace/79633e4e9fc4076e FAIL (2025-12-28 04:46:49 UTC) orbiter warning: context deadline exceeded Duration: 5.000950068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...XX...X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XX] duration: 5.000936191s trace: https://jaeger.dev.storj.io/trace/54b4291d2d6efd3e FAIL (2025-12-28 04:46:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000957051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [X.XXXXXXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X.] duration: 6.009731739s trace: https://jaeger.dev.storj.io/trace/78399034b5a785e0 FAIL (2025-12-28 04:46:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:45:44.734142397 (541h16m44.605225442s) [upload=474680]: Starting upload Dec 28 04:45:44.734224875 (541h16m44.60530792s) [upload=474680]: Done starting upload Dec 28 04:45:44.734367316 (541h16m44.605450351s) [upload=474680]: Uploading segments... Dec 28 04:45:44.734508254 (541h16m44.605591289s) [upload=474680]: Got next segment. Inline: false Dec 28 04:45:44.734551076 (541h16m44.605634111s) [upload=474680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:45:44.734570583 (541h16m44.605653618s) [upload=474680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:45:44.977050414 (541h16m44.848133459s) [seg_pos={0 0}, upload=474680]: Begin upload segment... Dec 28 04:45:44.977106280 (541h16m44.848189315s) [seg_pos={0 0}, upload=474680]: Done begin upload segment. Dec 28 04:45:44.977121670 (541h16m44.848204715s) [noise=true , address=45.91.116.8:4048 , piece_id=NJFEVWXIOBZ2EXRY2DKNMXB6JZRPVCPLS4S2NYUO4KGRL2CLPNYA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.977137048 (541h16m44.848220094s) [noise=true , address=145.79.227.15:4873 , piece_id=RKG5IGVNP6KHBTTVYVDSSWREBSFDNCMHFJ44CCQMVGJZKQLMAWXQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.977338673 (541h16m44.848421728s) [noise=true , address=45.91.116.6:4973 , piece_id=UGOYUOT3L2YU3WFGSZ3WP2JALZLLSF4MG5VCSIIZIQAQ3XPMAQ6A, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978028276 (541h16m44.849111321s) [noise=true , address=77.90.31.19:4844 , piece_id=IEFP4SXS44ER75RSZQEKDHFQ2L64LICEJVRWZXVUEGFZLZBENHHA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978423648 (541h16m44.849506693s) [noise=true , address=145.79.227.34:4905 , piece_id=FD6K5B32OKFWBROKPBPSJZ63R5NMGSIIHVD3RN5BSTK7V4TUTSFA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978878925 (541h16m44.84996197s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.979289828 (541h16m44.850372873s) [noise=true , address=77.90.31.21:4880 , piece_id=UMUGQUFOGTJ4YJPDKWUUJYMCOTUTH2DNDAMX2R4LN4GRVBHF4FCQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.979687725 (541h16m44.85077077s) [noise=true , address=77.90.31.13:4629 , piece_id=D55COMNZTRSQQGE7CHTMBX7DTWRZX3URAVSBGVFJAHNFZO6V4MKQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.980122744 (541h16m44.851205789s) [noise=true , address=45.91.116.11:4119 , piece_id=425WPUPDWNL3ACQ47D7LP2IJXVMN6TKWCXKHENIKX5W77WT7Q3BQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.981913293 (541h16m44.852996338s) [noise=true , address=145.79.227.51:4409 , piece_id=MWQLWVVRJMGWBMDGPY6ZKUD3GEGCI3GLA4L5NC3XQ6PCJSNGCTQA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.984102331 (541h16m44.855185367s) [noise=true , address=145.79.227.36:4518 , piece_id=YDCVMGJ4H54O44IPWXCNKMHS6GINDDF5M7TWZEBED53IUZEPV4YQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.986049640 (541h16m44.857132675s) [noise=true , address=145.79.227.47:4267 , piece_id=QCBZRWP2T7OR3Y2XV4FXGQKEYFTIFGS6CMAGBBGABKXMXAPTBM2A, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.987714420 (541h16m44.858797465s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.989278367 (541h16m44.860361412s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.990880558 (541h16m44.861963603s) [noise=true , address=145.79.227.10:4403 , piece_id=S2GXANYUM4MQX4LNAKUTH57MZZ4YN4Y3JJFOURJXPIVDKLNHHTHQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.992595845 (541h16m44.86367888s) [noise=true , address=145.79.227.3:4151 , piece_id=WB6DOLKIPKVIMX7J7JS72VEN5O2YNDH6H7GUFUWF3CJFIGRAMYWQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:46.704832736 (541h16m46.575915771s) [upload=474680]: Next returned nil segment Dec 28 04:45:46.704853466 (541h16m46.575936511s) [upload=474680]: Waiting for error group managing segments... Dec 28 04:45:46.983137443 (541h16m46.854220478s) [noise=true , address=145.79.227.15:4873 , piece_id=RKG5IGVNP6KHBTTVYVDSSWREBSFDNCMHFJ44CCQMVGJZKQLMAWXQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:46.996830186 (541h16m46.867913231s) [noise=true , address=145.79.227.36:4518 , piece_id=YDCVMGJ4H54O44IPWXCNKMHS6GINDDF5M7TWZEBED53IUZEPV4YQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:46.997882530 (541h16m46.868965575s) [noise=true , address=145.79.227.10:4403 , piece_id=S2GXANYUM4MQX4LNAKUTH57MZZ4YN4Y3JJFOURJXPIVDKLNHHTHQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.008211007 (541h16m46.879294052s) [noise=true , address=145.79.227.47:4267 , piece_id=QCBZRWP2T7OR3Y2XV4FXGQKEYFTIFGS6CMAGBBGABKXMXAPTBM2A, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.039492062 (541h16m46.910575097s) [noise=true , address=145.79.227.3:4151 , piece_id=WB6DOLKIPKVIMX7J7JS72VEN5O2YNDH6H7GUFUWF3CJFIGRAMYWQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.055717778 (541h16m46.926800823s) [noise=true , address=145.79.227.51:4409 , piece_id=MWQLWVVRJMGWBMDGPY6ZKUD3GEGCI3GLA4L5NC3XQ6PCJSNGCTQA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.267464488 (541h16m47.138547553s) [noise=true , address=77.90.31.13:4629 , piece_id=D55COMNZTRSQQGE7CHTMBX7DTWRZX3URAVSBGVFJAHNFZO6V4MKQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.273368700 (541h16m47.144451735s) [noise=true , address=45.91.116.6:4973 , piece_id=UGOYUOT3L2YU3WFGSZ3WP2JALZLLSF4MG5VCSIIZIQAQ3XPMAQ6A, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.279613912 (541h16m47.150696947s) [noise=true , address=45.91.116.11:4119 , piece_id=425WPUPDWNL3ACQ47D7LP2IJXVMN6TKWCXKHENIKX5W77WT7Q3BQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.400787305 (541h16m47.27187035s) [noise=true , address=45.91.116.8:4048 , piece_id=NJFEVWXIOBZ2EXRY2DKNMXB6JZRPVCPLS4S2NYUO4KGRL2CLPNYA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.259027010 (541h16m48.130110055s) [noise=true , address=145.79.227.34:4905 , piece_id=FD6K5B32OKFWBROKPBPSJZ63R5NMGSIIHVD3RN5BSTK7V4TUTSFA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.783117661 (541h16m48.654200696s) [noise=true , address=77.90.31.21:4880 , piece_id=UMUGQUFOGTJ4YJPDKWUUJYMCOTUTH2DNDAMX2R4LN4GRVBHF4FCQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.885156761 (541h16m48.756239796s) [noise=true , address=77.90.31.19:4844 , piece_id=IEFP4SXS44ER75RSZQEKDHFQ2L64LICEJVRWZXVUEGFZLZBENHHA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.885178291 (541h16m48.756261326s) [seg_pos={0 0}, upload=474680]: Segment reached optimal threshold of 13 pieces. Dec 28 04:45:48.885435412 (541h16m48.756518447s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44946->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:45:48.885442094 (541h16m48.756525139s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885512779 (541h16m48.756595814s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 04:45:48.885535542 (541h16m48.756618577s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885666361 (541h16m48.756749396s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:45:48.885672783 (541h16m48.756755818s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885687962 (541h16m48.756770997s) [seg_pos={0 0}, upload=474680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:45:48.885786730 (541h16m48.756869766s) [upload=474680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:45:48.885791449 (541h16m48.756874494s) [upload=474680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:45:49.175373480 (541h16m49.046456515s) [upload=474680]: Done uploading segments... Dec 28 04:45:49.175395492 (541h16m49.046478527s) [upload=474680]: Upload finished. err: <nil> Duration: 6.009750585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XX.XXXXXXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X] duration: 30.003266737s trace: https://jaeger.dev.storj.io/trace/78399034b5a785e0 FAIL (2025-12-28 04:46:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:45:44.734142397 (541h16m44.605225442s) [upload=474680]: Starting upload Dec 28 04:45:44.734224875 (541h16m44.60530792s) [upload=474680]: Done starting upload Dec 28 04:45:44.734367316 (541h16m44.605450351s) [upload=474680]: Uploading segments... Dec 28 04:45:44.734508254 (541h16m44.605591289s) [upload=474680]: Got next segment. Inline: false Dec 28 04:45:44.734551076 (541h16m44.605634111s) [upload=474680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:45:44.734570583 (541h16m44.605653618s) [upload=474680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:45:44.977050414 (541h16m44.848133459s) [seg_pos={0 0}, upload=474680]: Begin upload segment... Dec 28 04:45:44.977106280 (541h16m44.848189315s) [seg_pos={0 0}, upload=474680]: Done begin upload segment. Dec 28 04:45:44.977121670 (541h16m44.848204715s) [noise=true , address=45.91.116.8:4048 , piece_id=NJFEVWXIOBZ2EXRY2DKNMXB6JZRPVCPLS4S2NYUO4KGRL2CLPNYA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.977137048 (541h16m44.848220094s) [noise=true , address=145.79.227.15:4873 , piece_id=RKG5IGVNP6KHBTTVYVDSSWREBSFDNCMHFJ44CCQMVGJZKQLMAWXQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.977338673 (541h16m44.848421728s) [noise=true , address=45.91.116.6:4973 , piece_id=UGOYUOT3L2YU3WFGSZ3WP2JALZLLSF4MG5VCSIIZIQAQ3XPMAQ6A, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978028276 (541h16m44.849111321s) [noise=true , address=77.90.31.19:4844 , piece_id=IEFP4SXS44ER75RSZQEKDHFQ2L64LICEJVRWZXVUEGFZLZBENHHA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978423648 (541h16m44.849506693s) [noise=true , address=145.79.227.34:4905 , piece_id=FD6K5B32OKFWBROKPBPSJZ63R5NMGSIIHVD3RN5BSTK7V4TUTSFA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.978878925 (541h16m44.84996197s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.979289828 (541h16m44.850372873s) [noise=true , address=77.90.31.21:4880 , piece_id=UMUGQUFOGTJ4YJPDKWUUJYMCOTUTH2DNDAMX2R4LN4GRVBHF4FCQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.979687725 (541h16m44.85077077s) [noise=true , address=77.90.31.13:4629 , piece_id=D55COMNZTRSQQGE7CHTMBX7DTWRZX3URAVSBGVFJAHNFZO6V4MKQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.980122744 (541h16m44.851205789s) [noise=true , address=45.91.116.11:4119 , piece_id=425WPUPDWNL3ACQ47D7LP2IJXVMN6TKWCXKHENIKX5W77WT7Q3BQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.981913293 (541h16m44.852996338s) [noise=true , address=145.79.227.51:4409 , piece_id=MWQLWVVRJMGWBMDGPY6ZKUD3GEGCI3GLA4L5NC3XQ6PCJSNGCTQA, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.984102331 (541h16m44.855185367s) [noise=true , address=145.79.227.36:4518 , piece_id=YDCVMGJ4H54O44IPWXCNKMHS6GINDDF5M7TWZEBED53IUZEPV4YQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.986049640 (541h16m44.857132675s) [noise=true , address=145.79.227.47:4267 , piece_id=QCBZRWP2T7OR3Y2XV4FXGQKEYFTIFGS6CMAGBBGABKXMXAPTBM2A, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.987714420 (541h16m44.858797465s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.989278367 (541h16m44.860361412s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.990880558 (541h16m44.861963603s) [noise=true , address=145.79.227.10:4403 , piece_id=S2GXANYUM4MQX4LNAKUTH57MZZ4YN4Y3JJFOURJXPIVDKLNHHTHQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:44.992595845 (541h16m44.86367888s) [noise=true , address=145.79.227.3:4151 , piece_id=WB6DOLKIPKVIMX7J7JS72VEN5O2YNDH6H7GUFUWF3CJFIGRAMYWQ, seg_pos={0 0}, upload=474680]: Uploading piece... Dec 28 04:45:46.704832736 (541h16m46.575915771s) [upload=474680]: Next returned nil segment Dec 28 04:45:46.704853466 (541h16m46.575936511s) [upload=474680]: Waiting for error group managing segments... Dec 28 04:45:46.983137443 (541h16m46.854220478s) [noise=true , address=145.79.227.15:4873 , piece_id=RKG5IGVNP6KHBTTVYVDSSWREBSFDNCMHFJ44CCQMVGJZKQLMAWXQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:46.996830186 (541h16m46.867913231s) [noise=true , address=145.79.227.36:4518 , piece_id=YDCVMGJ4H54O44IPWXCNKMHS6GINDDF5M7TWZEBED53IUZEPV4YQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:46.997882530 (541h16m46.868965575s) [noise=true , address=145.79.227.10:4403 , piece_id=S2GXANYUM4MQX4LNAKUTH57MZZ4YN4Y3JJFOURJXPIVDKLNHHTHQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.008211007 (541h16m46.879294052s) [noise=true , address=145.79.227.47:4267 , piece_id=QCBZRWP2T7OR3Y2XV4FXGQKEYFTIFGS6CMAGBBGABKXMXAPTBM2A, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.039492062 (541h16m46.910575097s) [noise=true , address=145.79.227.3:4151 , piece_id=WB6DOLKIPKVIMX7J7JS72VEN5O2YNDH6H7GUFUWF3CJFIGRAMYWQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.055717778 (541h16m46.926800823s) [noise=true , address=145.79.227.51:4409 , piece_id=MWQLWVVRJMGWBMDGPY6ZKUD3GEGCI3GLA4L5NC3XQ6PCJSNGCTQA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.267464488 (541h16m47.138547553s) [noise=true , address=77.90.31.13:4629 , piece_id=D55COMNZTRSQQGE7CHTMBX7DTWRZX3URAVSBGVFJAHNFZO6V4MKQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.273368700 (541h16m47.144451735s) [noise=true , address=45.91.116.6:4973 , piece_id=UGOYUOT3L2YU3WFGSZ3WP2JALZLLSF4MG5VCSIIZIQAQ3XPMAQ6A, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.279613912 (541h16m47.150696947s) [noise=true , address=45.91.116.11:4119 , piece_id=425WPUPDWNL3ACQ47D7LP2IJXVMN6TKWCXKHENIKX5W77WT7Q3BQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:47.400787305 (541h16m47.27187035s) [noise=true , address=45.91.116.8:4048 , piece_id=NJFEVWXIOBZ2EXRY2DKNMXB6JZRPVCPLS4S2NYUO4KGRL2CLPNYA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.259027010 (541h16m48.130110055s) [noise=true , address=145.79.227.34:4905 , piece_id=FD6K5B32OKFWBROKPBPSJZ63R5NMGSIIHVD3RN5BSTK7V4TUTSFA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.783117661 (541h16m48.654200696s) [noise=true , address=77.90.31.21:4880 , piece_id=UMUGQUFOGTJ4YJPDKWUUJYMCOTUTH2DNDAMX2R4LN4GRVBHF4FCQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.885156761 (541h16m48.756239796s) [noise=true , address=77.90.31.19:4844 , piece_id=IEFP4SXS44ER75RSZQEKDHFQ2L64LICEJVRWZXVUEGFZLZBENHHA, seg_pos={0 0}, upload=474680]: Done uploading piece. err: <nil> Dec 28 04:45:48.885178291 (541h16m48.756261326s) [seg_pos={0 0}, upload=474680]: Segment reached optimal threshold of 13 pieces. Dec 28 04:45:48.885435412 (541h16m48.756518447s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:44946->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:45:48.885442094 (541h16m48.756525139s) [noise=true , address=45.91.116.14:4190 , piece_id=37MEEAHWTMAMBKHKDSFQDXUAXTVQH3R7Z2EAHDQ643GP5LWSWKKQ, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885512779 (541h16m48.756595814s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 04:45:48.885535542 (541h16m48.756618577s) [noise=true , address=77.90.31.18:4805 , piece_id=5NFC7YJ4IM5EHE3DTSLFQJWVODDNS3NPPUYGWZCRJ4BFUY6BKYHQ, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885666361 (541h16m48.756749396s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:45:48.885672783 (541h16m48.756755818s) [noise=true , address=145.79.227.44:4195 , piece_id=3UN4O3PT5QUJZM4ZTRTEKM6JERS2I4OPOYVVSODESFDQG62SZZ3Q, seg_pos={0 0}, upload=474680]: Long tail cancellation due to optimal threshold Dec 28 04:45:48.885687962 (541h16m48.756770997s) [seg_pos={0 0}, upload=474680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:45:48.885786730 (541h16m48.756869766s) [upload=474680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:45:48.885791449 (541h16m48.756874494s) [upload=474680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:45:49.175373480 (541h16m49.046456515s) [upload=474680]: Done uploading segments... Dec 28 04:45:49.175395492 (541h16m49.046478527s) [upload=474680]: Upload finished. err: <nil> Duration: 30.003288409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X......X.......................................] duration: 39.336318256s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:46:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 39.336338755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [....XX...X.X....X.X...XXXXX....XX..X.....XXX.XX.......X..X.X] duration: 5.001673854s trace: https://jaeger.dev.storj.io/trace/7eb769c6a3e6b7df FAIL (2025-12-28 04:45:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001748126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X......X......................................] duration: 54.770474457s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:45:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 54.770502971s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXX.XXXXXXX.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..] duration: 6.050897003s trace: https://jaeger.dev.storj.io/trace/6c88c881e14eb035 FAIL (2025-12-28 04:44:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: VHEEQMDDNQATK7P5QL2JLBHHONGHHFI4PIQSJJH5LAQK3LB53XYQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: FQNBS6ZIYANYJEGQDZZSPWUI2QHJM6IZ4472KBVSXNFZ4F2F4RWA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: PUETATJ6DX7YVYVWNXW5GLGEOD4LU2W3ONJK4ZVO33WHZ5E5LPRA) context deadline exceeded ======LOGS====== ments... Dec 28 04:43:31.513241415 (541h14m31.38432445s) [upload=474650]: Got next segment. Inline: false Dec 28 04:43:31.513598865 (541h14m31.38468191s) [upload=474650]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:43:31.513606008 (541h14m31.384689053s) [upload=474650]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:43:31.755585747 (541h14m31.626668782s) [seg_pos={0 0}, upload=474650]: Begin upload segment... Dec 28 04:43:31.755647154 (541h14m31.6267302s) [seg_pos={0 0}, upload=474650]: Done begin upload segment. Dec 28 04:43:31.755686309 (541h14m31.626769344s) [noise=true , address=145.79.227.46:4603 , piece_id=SYBHM5QEWJ2N6LMZ2ZE6SGNK2UO3OECNHIBXBUEAIT45K6KJKT5A, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.755752776 (541h14m31.626835821s) [noise=true , address=77.90.31.18:4808 , piece_id=VHEEQMDDNQATK7P5QL2JLBHHONGHHFI4PIQSJJH5LAQK3LB53XYQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.756375480 (541h14m31.627458515s) [noise=true , address=77.90.31.17:4773 , piece_id=S32VTXJYVATPB5SRTAZQ65T3BEE7ZFJWTV25E34JVKHGWFMSX3AQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.756859141 (541h14m31.627942187s) [noise=true , address=145.79.227.50:4373 , piece_id=P7LECCA6PVW24VSGZLXNSERPDNCD37BJM62G4POKQQ4M3OSA6B5A, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.756970404 (541h14m31.628053449s) [noise=true , address=45.91.116.14:4191 , piece_id=FQNBS6ZIYANYJEGQDZZSPWUI2QHJM6IZ4472KBVSXNFZ4F2F4RWA, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.757315560 (541h14m31.628398606s) [noise=true , address=145.79.227.45:4231 , piece_id=TMX2CHPECDGIPRAROONBBV4PFXRUFWM3KL6YQMVQI3C7ZGQNNKDQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.757350277 (541h14m31.628433322s) [noise=true , address=45.91.116.13:4154 , piece_id=PUETATJ6DX7YVYVWNXW5GLGEOD4LU2W3ONJK4ZVO33WHZ5E5LPRA, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.757728407 (541h14m31.628811452s) [noise=true , address=145.79.227.10:4401 , piece_id=ZBCWODVSDLIMRTQ35KLRJDBTCSL522D3VDIXDOCWWTD27WA7KOXA, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.757770127 (541h14m31.628853162s) [noise=true , address=145.79.227.17:4970 , piece_id=JTWACPE7ZIVB4LXN4JKU3UGRBQCKDF2GADZZKU6ANQRTPNNAZ7BQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.758079926 (541h14m31.629162971s) [noise=true , address=145.79.227.3:4150 , piece_id=ASWTMFSMHNWK3TJSN7GJR53KYARRMV7XUNU34YYAC2DD4DTNEPVQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.758313281 (541h14m31.629396326s) [noise=true , address=45.91.116.11:4119 , piece_id=BW5HWBVLHFV34AVX3ZC6QRNUPW4T2FSNRGIDFKDVLPBLIKXZHWIQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.758479207 (541h14m31.629562252s) [noise=true , address=145.79.227.15:4874 , piece_id=VX7OLFNUMEICTZOZQBPSLIDVICSPWRPZQZS3ENB7LONKBMGA5TYQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.759094639 (541h14m31.630177684s) [noise=true , address=145.79.227.16:4909 , piece_id=ETUEWK6IGVEZLSAX2NUPAX6HT7HGNIKTQ2IUV3JVD4FIJHLIXUOA, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.759633254 (541h14m31.630716299s) [noise=true , address=145.79.227.48:4303 , piece_id=DPGDZ6AI3LSYYOQAB4B5MNGMUVMCTFFXHQIUTSVH6SBPIWEO6FKQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.761978770 (541h14m31.633061815s) [noise=true , address=145.79.227.13:4802 , piece_id=35ODYNNP3B763YLN36KXNRPTOT4JAOHUMMCYXMBSVCM2ORZUHRZA, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:31.764017683 (541h14m31.635100728s) [noise=true , address=77.90.31.11:4556 , piece_id=KJPSW5BWZAH3FNYMIEHNMSK4OOMAYHFH7ROR26M5OQBRAOBZLOXQ, seg_pos={0 0}, upload=474650]: Uploading piece... Dec 28 04:43:33.588070672 (541h14m33.459153717s) [upload=474650]: Next returned nil segment Dec 28 04:43:33.588108123 (541h14m33.459191158s) [upload=474650]: Waiting for error group managing segments... Dec 28 04:43:33.948356193 (541h14m33.819439228s) [noise=true , address=77.90.31.11:4556 , piece_id=KJPSW5BWZAH3FNYMIEHNMSK4OOMAYHFH7ROR26M5OQBRAOBZLOXQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.036812538 (541h14m33.907895573s) [noise=true , address=77.90.31.17:4773 , piece_id=S32VTXJYVATPB5SRTAZQ65T3BEE7ZFJWTV25E34JVKHGWFMSX3AQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.130037102 (541h14m34.001120137s) [noise=true , address=77.90.31.18:4808 , piece_id=VHEEQMDDNQATK7P5QL2JLBHHONGHHFI4PIQSJJH5LAQK3LB53XYQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.324759825 (541h14m34.19584286s) [noise=true , address=45.91.116.11:4119 , piece_id=BW5HWBVLHFV34AVX3ZC6QRNUPW4T2FSNRGIDFKDVLPBLIKXZHWIQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.451777928 (541h14m34.322860963s) [noise=true , address=145.79.227.16:4909 , piece_id=ETUEWK6IGVEZLSAX2NUPAX6HT7HGNIKTQ2IUV3JVD4FIJHLIXUOA, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.454566949 (541h14m34.325649994s) [noise=true , address=145.79.227.17:4970 , piece_id=JTWACPE7ZIVB4LXN4JKU3UGRBQCKDF2GADZZKU6ANQRTPNNAZ7BQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.458926871 (541h14m34.330009906s) [noise=true , address=145.79.227.10:4401 , piece_id=ZBCWODVSDLIMRTQ35KLRJDBTCSL522D3VDIXDOCWWTD27WA7KOXA, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.499562277 (541h14m34.370645312s) [noise=true , address=145.79.227.15:4874 , piece_id=VX7OLFNUMEICTZOZQBPSLIDVICSPWRPZQZS3ENB7LONKBMGA5TYQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.530935917 (541h14m34.402018952s) [noise=true , address=145.79.227.50:4373 , piece_id=P7LECCA6PVW24VSGZLXNSERPDNCD37BJM62G4POKQQ4M3OSA6B5A, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:34.685965349 (541h14m34.557048384s) [noise=true , address=45.91.116.14:4191 , piece_id=FQNBS6ZIYANYJEGQDZZSPWUI2QHJM6IZ4472KBVSXNFZ4F2F4RWA, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:35.171695977 (541h14m35.042779012s) [noise=true , address=145.79.227.46:4603 , piece_id=SYBHM5QEWJ2N6LMZ2ZE6SGNK2UO3OECNHIBXBUEAIT45K6KJKT5A, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:35.497961619 (541h14m35.369044664s) [noise=true , address=45.91.116.13:4154 , piece_id=PUETATJ6DX7YVYVWNXW5GLGEOD4LU2W3ONJK4ZVO33WHZ5E5LPRA, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:57.161143263 (541h14m57.032226308s) [noise=true , address=145.79.227.48:4303 , piece_id=DPGDZ6AI3LSYYOQAB4B5MNGMUVMCTFFXHQIUTSVH6SBPIWEO6FKQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: <nil> Dec 28 04:43:57.161165495 (541h14m57.032248531s) [seg_pos={0 0}, upload=474650]: Segment reached optimal threshold of 13 pieces. Dec 28 04:43:57.161334777 (541h14m57.032417822s) [noise=true , address=145.79.227.13:4802 , piece_id=35ODYNNP3B763YLN36KXNRPTOT4JAOHUMMCYXMBSVCM2ORZUHRZA, seg_pos={0 0}, upload=474650]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:42168->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:42168->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42168->145.79.227.13:4802: use of closed network connection Dec 28 04:43:57.161340519 (541h14m57.032423564s) [noise=true , address=145.79.227.13:4802 , piece_id=35ODYNNP3B763YLN36KXNRPTOT4JAOHUMMCYXMBSVCM2ORZUHRZA, seg_pos={0 0}, upload=474650]: Long tail cancellation due to optimal threshold Dec 28 04:43:57.161466408 (541h14m57.032549453s) [noise=true , address=145.79.227.3:4150 , piece_id=ASWTMFSMHNWK3TJSN7GJR53KYARRMV7XUNU34YYAC2DD4DTNEPVQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Dec 28 04:43:57.161472129 (541h14m57.032555164s) [noise=true , address=145.79.227.3:4150 , piece_id=ASWTMFSMHNWK3TJSN7GJR53KYARRMV7XUNU34YYAC2DD4DTNEPVQ, seg_pos={0 0}, upload=474650]: Long tail cancellation due to optimal threshold Dec 28 04:43:57.161553594 (541h14m57.032636629s) [noise=true , address=145.79.227.45:4231 , piece_id=TMX2CHPECDGIPRAROONBBV4PFXRUFWM3KL6YQMVQI3C7ZGQNNKDQ, seg_pos={0 0}, upload=474650]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: write tcp 188.166.225.170:54820->145.79.227.45:4231: use of closed network connection; write tcp 188.166.225.170:54820->145.79.227.45:4231: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54820->145.79.227.45:4231: use of closed network connection Dec 28 04:43:57.161577349 (541h14m57.032660434s) [noise=true , address=145.79.227.45:4231 , piece_id=TMX2CHPECDGIPRAROONBBV4PFXRUFWM3KL6YQMVQI3C7ZGQNNKDQ, seg_pos={0 0}, upload=474650]: Long tail cancellation due to optimal threshold Dec 28 04:43:57.161634688 (541h14m57.032717734s) [seg_pos={0 0}, upload=474650]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:43:57.161716714 (541h14m57.032799749s) [upload=474650]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:43:57.161721374 (541h14m57.032804409s) [upload=474650]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:43:57.440571198 (541h14m57.311654233s) [upload=474650]: Done uploading segments... Dec 28 04:43:57.440593641 (541h14m57.311676676s) [upload=474650]: Upload finished. err: <nil> Duration: 6.050923434s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)