Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 907023 reports (Page 11245 of 18141)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXXXX.XXX.X] duration: 5.000583195s trace: https://jaeger.dev.storj.io/trace/5d17eab3041af0db FAIL (2025-12-30 13:30:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000624485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X...........X..X........X...X...XXX.......X...XX.XX..X.X] duration: 30.001365913s trace: https://jaeger.dev.storj.io/trace/29143bf0b16252c7 FAIL (2025-12-30 13:30:42 UTC) orbiter warning: context deadline exceeded Duration: 30.001393546s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X..X.X..X....X........XX................X.XX....XX..XXX.] duration: 5.950960286s trace: https://jaeger.dev.storj.io/trace/2edad4b1c28a2605 FAIL (2025-12-30 13:30:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: S2ECZB5BK5GZ2FPWWYAGEKCPS6HGYZMAWQMZ2XP5CQHZEHAC6YNQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VH2OSOXNTWBUWDEVCGARDIXFQG2GGK6VIYEPJFW5GDVPA6DP4JLQ) context deadline exceeded ======LOGS====== Dec 30 13:29:48.510061130 (598h0m48.381144175s) [upload=524186]: Starting upload Dec 30 13:29:48.510123910 (598h0m48.381206955s) [upload=524186]: Done starting upload Dec 30 13:29:48.510213049 (598h0m48.381296094s) [upload=524186]: Uploading segments... Dec 30 13:29:48.510296278 (598h0m48.381379313s) [upload=524186]: Got next segment. Inline: false Dec 30 13:29:48.510335212 (598h0m48.381418247s) [upload=524186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:29:48.510339290 (598h0m48.381422325s) [upload=524186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:29:48.757051807 (598h0m48.628134862s) [seg_pos={0 0}, upload=524186]: Begin upload segment... Dec 30 13:29:48.757125577 (598h0m48.628208612s) [seg_pos={0 0}, upload=524186]: Done begin upload segment. Dec 30 13:29:48.757183568 (598h0m48.628266613s) [noise=true , address=145.79.227.10:4402 , piece_id=ERVSAEANWPLGVBASN5US5KGKGHXNJQ6FLIBDJNK25PGCNIMZ4TAA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.757199798 (598h0m48.628282833s) [noise=true , address=145.79.227.34:4907 , piece_id=5Z5P5OMNUSWESCWSHCU2H3CQ2NFZAK3BCTUSVSB7W7HSHWRQD2FQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.757918648 (598h0m48.629001693s) [noise=true , address=45.91.116.10:4081 , piece_id=VH2OSOXNTWBUWDEVCGARDIXFQG2GGK6VIYEPJFW5GDVPA6DP4JLQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.758262131 (598h0m48.629345166s) [noise=true , address=77.90.31.10:4519 , piece_id=45CIDDMZADT4Z37ZAIFFXOMHS2UI4EMGZBMD5CPCEYMBP5BN25YA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.758532085 (598h0m48.62961512s) [noise=true , address=45.91.116.7:4009 , piece_id=UGTFSWGSOX2J47RQBQ5E2AW3OJ6LOM4LAZSJQIDNR4XXZQDRNFMQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.758804203 (598h0m48.629887248s) [noise=true , address=145.79.227.45:4230 , piece_id=BUBJATRSFGCXLQPKKKTKZOYRFJQFT2IGOSWAEQMBL3B666MVEYUA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.759116698 (598h0m48.630199744s) [noise=true , address=145.79.227.14:4840 , piece_id=IKNFVJDSXVWMFTX7UJIFW7MGWVBS7TW5SWCDD57GLXMR5DC3WWEA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.759527711 (598h0m48.630610746s) [noise=true , address=45.91.116.14:4190 , piece_id=MXHJOENBZQGJSYQOO72VF4C4HBJ2MFVKIFUWNIU6Q7Q3NLSZGZZA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.759925899 (598h0m48.631008934s) [noise=true , address=77.90.31.11:4554 , piece_id=62UG6C3NINS5ABHFHNC5RO5RGW33DWOF2N4Z3RU777ZPNBC6F2MA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.761608142 (598h0m48.632691197s) [noise=true , address=145.79.227.12:4767 , piece_id=37JNE22EDDJPY3IV3UJVH2EQCKJTRL22TXOBXUHBJPBKXCHQHYRQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.762101712 (598h0m48.633184757s) [noise=true , address=145.79.227.42:4121 , piece_id=YRI73ZTGXNIMKDSH4U5RPG54QX22LWOVLQDSGIODXFCGE5JJOP7Q, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.762527313 (598h0m48.633610358s) [noise=true , address=45.91.116.11:4119 , piece_id=S2ECZB5BK5GZ2FPWWYAGEKCPS6HGYZMAWQMZ2XP5CQHZEHAC6YNQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.763005623 (598h0m48.634088668s) [noise=true , address=145.79.227.19:4079 , piece_id=Q4UJNFPGAXM4ZR5RSWJXAL274GPB554ITMXIWGVS7AGFECZT6ITQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.764540896 (598h0m48.635623941s) [noise=true , address=45.91.116.8:4046 , piece_id=Z6D4CN77TNV622XM37O52MMADSSUBF6MF3RHX2LJRZFQX4GRVDYQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.766011978 (598h0m48.637095023s) [noise=true , address=145.79.227.8:4330 , piece_id=IIAGN7DI6P4NXW3OXVXHZP2WOZNSRFA5ABZ6V4DZR5W6FOKCBRFA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.767445207 (598h0m48.638528252s) [noise=true , address=77.90.31.18:4808 , piece_id=3MAXSIVZGWRORICSKQRBDHHLFUE7HN74S7SNPAYTWXAETGKW2OIA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:50.456763789 (598h0m50.327846834s) [upload=524186]: Next returned nil segment Dec 30 13:29:50.456783847 (598h0m50.327866892s) [upload=524186]: Waiting for error group managing segments... Dec 30 13:29:50.746222291 (598h0m50.617305336s) [noise=true , address=145.79.227.10:4402 , piece_id=ERVSAEANWPLGVBASN5US5KGKGHXNJQ6FLIBDJNK25PGCNIMZ4TAA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.755037977 (598h0m50.626121022s) [noise=true , address=145.79.227.42:4121 , piece_id=YRI73ZTGXNIMKDSH4U5RPG54QX22LWOVLQDSGIODXFCGE5JJOP7Q, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.764599473 (598h0m50.635682508s) [noise=true , address=145.79.227.14:4840 , piece_id=IKNFVJDSXVWMFTX7UJIFW7MGWVBS7TW5SWCDD57GLXMR5DC3WWEA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.769708421 (598h0m50.640791457s) [noise=true , address=145.79.227.8:4330 , piece_id=IIAGN7DI6P4NXW3OXVXHZP2WOZNSRFA5ABZ6V4DZR5W6FOKCBRFA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.772189466 (598h0m50.643272501s) [noise=true , address=145.79.227.45:4230 , piece_id=BUBJATRSFGCXLQPKKKTKZOYRFJQFT2IGOSWAEQMBL3B666MVEYUA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.778082978 (598h0m50.649166023s) [noise=true , address=145.79.227.19:4079 , piece_id=Q4UJNFPGAXM4ZR5RSWJXAL274GPB554ITMXIWGVS7AGFECZT6ITQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.823829796 (598h0m50.694912831s) [noise=true , address=145.79.227.12:4767 , piece_id=37JNE22EDDJPY3IV3UJVH2EQCKJTRL22TXOBXUHBJPBKXCHQHYRQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.023801773 (598h0m50.894884808s) [noise=true , address=77.90.31.10:4519 , piece_id=45CIDDMZADT4Z37ZAIFFXOMHS2UI4EMGZBMD5CPCEYMBP5BN25YA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.082438256 (598h0m50.953521291s) [noise=true , address=45.91.116.11:4119 , piece_id=S2ECZB5BK5GZ2FPWWYAGEKCPS6HGYZMAWQMZ2XP5CQHZEHAC6YNQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.093479409 (598h0m50.964562455s) [noise=true , address=45.91.116.8:4046 , piece_id=Z6D4CN77TNV622XM37O52MMADSSUBF6MF3RHX2LJRZFQX4GRVDYQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.160420515 (598h0m51.03150355s) [noise=true , address=45.91.116.10:4081 , piece_id=VH2OSOXNTWBUWDEVCGARDIXFQG2GGK6VIYEPJFW5GDVPA6DP4JLQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.695174036 (598h0m51.566257071s) [noise=true , address=77.90.31.11:4554 , piece_id=62UG6C3NINS5ABHFHNC5RO5RGW33DWOF2N4Z3RU777ZPNBC6F2MA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.858160387 (598h0m51.729243432s) [noise=true , address=77.90.31.18:4808 , piece_id=3MAXSIVZGWRORICSKQRBDHHLFUE7HN74S7SNPAYTWXAETGKW2OIA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.858185576 (598h0m51.729268621s) [seg_pos={0 0}, upload=524186]: Segment reached optimal threshold of 13 pieces. Dec 30 13:29:51.858389484 (598h0m51.729472519s) [noise=true , address=45.91.116.14:4190 , piece_id=MXHJOENBZQGJSYQOO72VF4C4HBJ2MFVKIFUWNIU6Q7Q3NLSZGZZA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:45648->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:45648->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45648->45.91.116.14:4190: use of closed network connection Dec 30 13:29:51.858396808 (598h0m51.729479853s) [noise=true , address=45.91.116.14:4190 , piece_id=MXHJOENBZQGJSYQOO72VF4C4HBJ2MFVKIFUWNIU6Q7Q3NLSZGZZA, seg_pos={0 0}, upload=524186]: Long tail cancellation due to optimal threshold Dec 30 13:29:51.858432846 (598h0m51.729515891s) [noise=true , address=45.91.116.7:4009 , piece_id=UGTFSWGSOX2J47RQBQ5E2AW3OJ6LOM4LAZSJQIDNR4XXZQDRNFMQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 13:29:51.858442796 (598h0m51.729525841s) [noise=true , address=45.91.116.7:4009 , piece_id=UGTFSWGSOX2J47RQBQ5E2AW3OJ6LOM4LAZSJQIDNR4XXZQDRNFMQ, seg_pos={0 0}, upload=524186]: Long tail cancellation due to optimal threshold Dec 30 13:29:51.858611326 (598h0m51.729694381s) [noise=true , address=145.79.227.34:4907 , piece_id=5Z5P5OMNUSWESCWSHCU2H3CQ2NFZAK3BCTUSVSB7W7HSHWRQD2FQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 13:29:51.858619252 (598h0m51.729702297s) [noise=true , address=145.79.227.34:4907 , piece_id=5Z5P5OMNUSWESCWSHCU2H3CQ2NFZAK3BCTUSVSB7W7HSHWRQD2FQ, seg_pos={0 0}, upload=524186]: Long tail cancellation due to optimal threshold Dec 30 13:29:51.858647244 (598h0m51.729730279s) [seg_pos={0 0}, upload=524186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:29:51.858739500 (598h0m51.729822545s) [upload=524186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:29:51.858745632 (598h0m51.729828677s) [upload=524186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:29:52.154575075 (598h0m52.02565811s) [upload=524186]: Done uploading segments... Dec 30 13:29:52.154599692 (598h0m52.025682737s) [upload=524186]: Upload finished. err: <nil> Duration: 5.950975836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X..X.X..X....X........XX................X.XX....XX..XXX] duration: 30.004593292s trace: https://jaeger.dev.storj.io/trace/2edad4b1c28a2605 FAIL (2025-12-30 13:30:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: S2ECZB5BK5GZ2FPWWYAGEKCPS6HGYZMAWQMZ2XP5CQHZEHAC6YNQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VH2OSOXNTWBUWDEVCGARDIXFQG2GGK6VIYEPJFW5GDVPA6DP4JLQ) context deadline exceeded ======LOGS====== Dec 30 13:29:48.510061130 (598h0m48.381144175s) [upload=524186]: Starting upload Dec 30 13:29:48.510123910 (598h0m48.381206955s) [upload=524186]: Done starting upload Dec 30 13:29:48.510213049 (598h0m48.381296094s) [upload=524186]: Uploading segments... Dec 30 13:29:48.510296278 (598h0m48.381379313s) [upload=524186]: Got next segment. Inline: false Dec 30 13:29:48.510335212 (598h0m48.381418247s) [upload=524186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:29:48.510339290 (598h0m48.381422325s) [upload=524186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:29:48.757051807 (598h0m48.628134862s) [seg_pos={0 0}, upload=524186]: Begin upload segment... Dec 30 13:29:48.757125577 (598h0m48.628208612s) [seg_pos={0 0}, upload=524186]: Done begin upload segment. Dec 30 13:29:48.757183568 (598h0m48.628266613s) [noise=true , address=145.79.227.10:4402 , piece_id=ERVSAEANWPLGVBASN5US5KGKGHXNJQ6FLIBDJNK25PGCNIMZ4TAA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.757199798 (598h0m48.628282833s) [noise=true , address=145.79.227.34:4907 , piece_id=5Z5P5OMNUSWESCWSHCU2H3CQ2NFZAK3BCTUSVSB7W7HSHWRQD2FQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.757918648 (598h0m48.629001693s) [noise=true , address=45.91.116.10:4081 , piece_id=VH2OSOXNTWBUWDEVCGARDIXFQG2GGK6VIYEPJFW5GDVPA6DP4JLQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.758262131 (598h0m48.629345166s) [noise=true , address=77.90.31.10:4519 , piece_id=45CIDDMZADT4Z37ZAIFFXOMHS2UI4EMGZBMD5CPCEYMBP5BN25YA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.758532085 (598h0m48.62961512s) [noise=true , address=45.91.116.7:4009 , piece_id=UGTFSWGSOX2J47RQBQ5E2AW3OJ6LOM4LAZSJQIDNR4XXZQDRNFMQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.758804203 (598h0m48.629887248s) [noise=true , address=145.79.227.45:4230 , piece_id=BUBJATRSFGCXLQPKKKTKZOYRFJQFT2IGOSWAEQMBL3B666MVEYUA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.759116698 (598h0m48.630199744s) [noise=true , address=145.79.227.14:4840 , piece_id=IKNFVJDSXVWMFTX7UJIFW7MGWVBS7TW5SWCDD57GLXMR5DC3WWEA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.759527711 (598h0m48.630610746s) [noise=true , address=45.91.116.14:4190 , piece_id=MXHJOENBZQGJSYQOO72VF4C4HBJ2MFVKIFUWNIU6Q7Q3NLSZGZZA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.759925899 (598h0m48.631008934s) [noise=true , address=77.90.31.11:4554 , piece_id=62UG6C3NINS5ABHFHNC5RO5RGW33DWOF2N4Z3RU777ZPNBC6F2MA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.761608142 (598h0m48.632691197s) [noise=true , address=145.79.227.12:4767 , piece_id=37JNE22EDDJPY3IV3UJVH2EQCKJTRL22TXOBXUHBJPBKXCHQHYRQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.762101712 (598h0m48.633184757s) [noise=true , address=145.79.227.42:4121 , piece_id=YRI73ZTGXNIMKDSH4U5RPG54QX22LWOVLQDSGIODXFCGE5JJOP7Q, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.762527313 (598h0m48.633610358s) [noise=true , address=45.91.116.11:4119 , piece_id=S2ECZB5BK5GZ2FPWWYAGEKCPS6HGYZMAWQMZ2XP5CQHZEHAC6YNQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.763005623 (598h0m48.634088668s) [noise=true , address=145.79.227.19:4079 , piece_id=Q4UJNFPGAXM4ZR5RSWJXAL274GPB554ITMXIWGVS7AGFECZT6ITQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.764540896 (598h0m48.635623941s) [noise=true , address=45.91.116.8:4046 , piece_id=Z6D4CN77TNV622XM37O52MMADSSUBF6MF3RHX2LJRZFQX4GRVDYQ, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.766011978 (598h0m48.637095023s) [noise=true , address=145.79.227.8:4330 , piece_id=IIAGN7DI6P4NXW3OXVXHZP2WOZNSRFA5ABZ6V4DZR5W6FOKCBRFA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:48.767445207 (598h0m48.638528252s) [noise=true , address=77.90.31.18:4808 , piece_id=3MAXSIVZGWRORICSKQRBDHHLFUE7HN74S7SNPAYTWXAETGKW2OIA, seg_pos={0 0}, upload=524186]: Uploading piece... Dec 30 13:29:50.456763789 (598h0m50.327846834s) [upload=524186]: Next returned nil segment Dec 30 13:29:50.456783847 (598h0m50.327866892s) [upload=524186]: Waiting for error group managing segments... Dec 30 13:29:50.746222291 (598h0m50.617305336s) [noise=true , address=145.79.227.10:4402 , piece_id=ERVSAEANWPLGVBASN5US5KGKGHXNJQ6FLIBDJNK25PGCNIMZ4TAA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.755037977 (598h0m50.626121022s) [noise=true , address=145.79.227.42:4121 , piece_id=YRI73ZTGXNIMKDSH4U5RPG54QX22LWOVLQDSGIODXFCGE5JJOP7Q, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.764599473 (598h0m50.635682508s) [noise=true , address=145.79.227.14:4840 , piece_id=IKNFVJDSXVWMFTX7UJIFW7MGWVBS7TW5SWCDD57GLXMR5DC3WWEA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.769708421 (598h0m50.640791457s) [noise=true , address=145.79.227.8:4330 , piece_id=IIAGN7DI6P4NXW3OXVXHZP2WOZNSRFA5ABZ6V4DZR5W6FOKCBRFA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.772189466 (598h0m50.643272501s) [noise=true , address=145.79.227.45:4230 , piece_id=BUBJATRSFGCXLQPKKKTKZOYRFJQFT2IGOSWAEQMBL3B666MVEYUA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.778082978 (598h0m50.649166023s) [noise=true , address=145.79.227.19:4079 , piece_id=Q4UJNFPGAXM4ZR5RSWJXAL274GPB554ITMXIWGVS7AGFECZT6ITQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:50.823829796 (598h0m50.694912831s) [noise=true , address=145.79.227.12:4767 , piece_id=37JNE22EDDJPY3IV3UJVH2EQCKJTRL22TXOBXUHBJPBKXCHQHYRQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.023801773 (598h0m50.894884808s) [noise=true , address=77.90.31.10:4519 , piece_id=45CIDDMZADT4Z37ZAIFFXOMHS2UI4EMGZBMD5CPCEYMBP5BN25YA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.082438256 (598h0m50.953521291s) [noise=true , address=45.91.116.11:4119 , piece_id=S2ECZB5BK5GZ2FPWWYAGEKCPS6HGYZMAWQMZ2XP5CQHZEHAC6YNQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.093479409 (598h0m50.964562455s) [noise=true , address=45.91.116.8:4046 , piece_id=Z6D4CN77TNV622XM37O52MMADSSUBF6MF3RHX2LJRZFQX4GRVDYQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.160420515 (598h0m51.03150355s) [noise=true , address=45.91.116.10:4081 , piece_id=VH2OSOXNTWBUWDEVCGARDIXFQG2GGK6VIYEPJFW5GDVPA6DP4JLQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.695174036 (598h0m51.566257071s) [noise=true , address=77.90.31.11:4554 , piece_id=62UG6C3NINS5ABHFHNC5RO5RGW33DWOF2N4Z3RU777ZPNBC6F2MA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.858160387 (598h0m51.729243432s) [noise=true , address=77.90.31.18:4808 , piece_id=3MAXSIVZGWRORICSKQRBDHHLFUE7HN74S7SNPAYTWXAETGKW2OIA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: <nil> Dec 30 13:29:51.858185576 (598h0m51.729268621s) [seg_pos={0 0}, upload=524186]: Segment reached optimal threshold of 13 pieces. Dec 30 13:29:51.858389484 (598h0m51.729472519s) [noise=true , address=45.91.116.14:4190 , piece_id=MXHJOENBZQGJSYQOO72VF4C4HBJ2MFVKIFUWNIU6Q7Q3NLSZGZZA, seg_pos={0 0}, upload=524186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:45648->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:45648->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45648->45.91.116.14:4190: use of closed network connection Dec 30 13:29:51.858396808 (598h0m51.729479853s) [noise=true , address=45.91.116.14:4190 , piece_id=MXHJOENBZQGJSYQOO72VF4C4HBJ2MFVKIFUWNIU6Q7Q3NLSZGZZA, seg_pos={0 0}, upload=524186]: Long tail cancellation due to optimal threshold Dec 30 13:29:51.858432846 (598h0m51.729515891s) [noise=true , address=45.91.116.7:4009 , piece_id=UGTFSWGSOX2J47RQBQ5E2AW3OJ6LOM4LAZSJQIDNR4XXZQDRNFMQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 13:29:51.858442796 (598h0m51.729525841s) [noise=true , address=45.91.116.7:4009 , piece_id=UGTFSWGSOX2J47RQBQ5E2AW3OJ6LOM4LAZSJQIDNR4XXZQDRNFMQ, seg_pos={0 0}, upload=524186]: Long tail cancellation due to optimal threshold Dec 30 13:29:51.858611326 (598h0m51.729694381s) [noise=true , address=145.79.227.34:4907 , piece_id=5Z5P5OMNUSWESCWSHCU2H3CQ2NFZAK3BCTUSVSB7W7HSHWRQD2FQ, seg_pos={0 0}, upload=524186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 13:29:51.858619252 (598h0m51.729702297s) [noise=true , address=145.79.227.34:4907 , piece_id=5Z5P5OMNUSWESCWSHCU2H3CQ2NFZAK3BCTUSVSB7W7HSHWRQD2FQ, seg_pos={0 0}, upload=524186]: Long tail cancellation due to optimal threshold Dec 30 13:29:51.858647244 (598h0m51.729730279s) [seg_pos={0 0}, upload=524186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:29:51.858739500 (598h0m51.729822545s) [upload=524186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:29:51.858745632 (598h0m51.729828677s) [upload=524186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:29:52.154575075 (598h0m52.02565811s) [upload=524186]: Done uploading segments... Dec 30 13:29:52.154599692 (598h0m52.025682737s) [upload=524186]: Upload finished. err: <nil> Duration: 30.004617408s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X...........X..X........X...X...XXX.......X...XX.XX..X.] duration: 18.42249193s trace: https://jaeger.dev.storj.io/trace/74b25e1c529adfc4 FAIL (2025-12-30 13:29:59 UTC) orbiter warning: context deadline exceeded Duration: 18.422519082s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXXXX.XXX.] duration: 3.775461278s trace: https://jaeger.dev.storj.io/trace/454caae2fb5c354f FAIL (2025-12-30 13:29:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.775488208s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X...........X..X........X...X...XXX.......X...XX.XX..X] duration: 30.001467222s trace: https://jaeger.dev.storj.io/trace/74b25e1c529adfc4 FAIL (2025-12-30 13:29:36 UTC) orbiter warning: context deadline exceeded Duration: 30.001489454s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X....X........X..............X.......X...X..............X.X.] duration: 15.373989186s trace: https://jaeger.dev.storj.io/trace/215247d8c89b6452 FAIL (2025-12-30 13:29:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments... Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces. Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments... Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil> Duration: 15.374016669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration: 13.753177033s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:29:32 UTC) orbiter warning: context deadline exceeded Duration: 13.753207369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X..X.X..X....X........XX................X.XX....XX..XX] duration: 30.003583758s trace: https://jaeger.dev.storj.io/trace/4fded8ee44f0e304 FAIL (2025-12-30 13:29:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: D36STZV6XH6CHIH2EM5KE66DYJGYDKP6XYSCLTGPVWEANESOM3UQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: K3VJPV7MQXIXB5HDHTZMRKGDM6KDLAUAI2IVFU5TU6Y7LCXEXSFQ) context deadline exceeded ======LOGS====== Dec 30 13:28:55.716344132 (597h59m55.587427177s) [upload=524174]: Starting upload Dec 30 13:28:55.716395741 (597h59m55.587478776s) [upload=524174]: Done starting upload Dec 30 13:28:55.716590712 (597h59m55.587673757s) [upload=524174]: Uploading segments... Dec 30 13:28:55.716639936 (597h59m55.587722971s) [upload=524174]: Got next segment. Inline: false Dec 30 13:28:55.716668210 (597h59m55.587751245s) [upload=524174]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:28:55.716672779 (597h59m55.587755824s) [upload=524174]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:28:55.962420228 (597h59m55.833503263s) [seg_pos={0 0}, upload=524174]: Begin upload segment... Dec 30 13:28:55.962493808 (597h59m55.833576843s) [seg_pos={0 0}, upload=524174]: Done begin upload segment. Dec 30 13:28:55.962535517 (597h59m55.833618562s) [noise=true , address=145.79.227.24:4581 , piece_id=RNQYOP2XOETR3RIO2SXUXWBDPIA7XUAF6PZSQZ6UU3LRVBFWGPCQ, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.962660225 (597h59m55.83374326s) [noise=true , address=77.90.31.10:4520 , piece_id=K3VJPV7MQXIXB5HDHTZMRKGDM6KDLAUAI2IVFU5TU6Y7LCXEXSFQ, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.963754548 (597h59m55.834837583s) [noise=true , address=145.79.227.25:4617 , piece_id=4SX4DV7H5PBGYE2JXQDNJCXJFZHBEMDWMBH623CFM3DGJ2BYWPVQ, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.964073667 (597h59m55.835156702s) [noise=true , address=77.90.31.11:4553 , piece_id=NPAJEEBODUMLAPQ6DXSBXN7BQP7A4TEDR3PE52GN34DJ53AYIQ5Q, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.964908887 (597h59m55.835991922s) [noise=true , address=145.79.227.45:4231 , piece_id=RGSCYHMMLSATNC7FSB6JPGCJPRPJKBC45WWYEQFLFS7LJR6U2EYQ, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.965412045 (597h59m55.83649509s) [noise=true , address=145.79.227.7:4295 , piece_id=HINKDYRST6Z6QTC7BUYL2NX64UIVIFXZG2HZVUBLQ3YPRZSO47UA, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.966396970 (597h59m55.837480005s) [noise=true , address=145.79.227.26:4653 , piece_id=OIYDLLKF3VKICZHWUPANYXHKZ6UPPJV3LL6IT5KTR6W2YD7LHWCA, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.966708263 (597h59m55.837791298s) [noise=true , address=145.79.227.2:4115 , piece_id=7TKKPHCDVVHSUBUAJYA2CDMI4AYGUC5GSGLZVH246KU3GFUZ3IPQ, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.967643394 (597h59m55.838726429s) [noise=true , address=145.79.227.38:4978 , piece_id=2NJCYSCE547YO3QUQLXYRG5YXBCCNSRSADI4NW4AVUAWDA5UWTQA, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.968087760 (597h59m55.839170795s) [noise=true , address=45.91.116.5:4300 , piece_id=NFBCKO3TO7CENHFR7H4UYTT3R4QKZZA6WMG6BNREUFODODVKK5AA, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.968883826 (597h59m55.839966861s) [noise=true , address=145.79.227.3:4150 , piece_id=PPOHFUDOBD5G4MGNKB3N7CJI64AILUWKR4HVFATHKENWSYJ3UD6A, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.969237078 (597h59m55.840320113s) [noise=true , address=45.91.116.8:4046 , piece_id=D36STZV6XH6CHIH2EM5KE66DYJGYDKP6XYSCLTGPVWEANESOM3UQ, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.969573099 (597h59m55.840656264s) [noise=true , address=145.79.227.46:4603 , piece_id=5FKP2DLPZS4IQIYGKTS7KEGWIJHZMGDDZMOB4XPMDOZIVKEEAFOA, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.970003608 (597h59m55.841086643s) [noise=true , address=145.79.227.44:4195 , piece_id=J4QNUZ7CN44TVECPGHG75PD2TP2P7AUWSSXYR4UULSFCPSDHNJRQ, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.970451341 (597h59m55.841534376s) [noise=true , address=145.79.227.8:4331 , piece_id=UTEN5C5VDDGF6X4CLX7P5V336RUGY7AQADV47PMIAI4MVQZWKOQA, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:55.971787224 (597h59m55.842870259s) [noise=true , address=145.79.227.15:4873 , piece_id=QEN7EXHJY22WHEHAS544KWQWUKJ44G4SLHUECPOUAQOESXX2YTKA, seg_pos={0 0}, upload=524174]: Uploading piece... Dec 30 13:28:57.579282863 (597h59m57.450365908s) [upload=524174]: Next returned nil segment Dec 30 13:28:57.579302841 (597h59m57.450385876s) [upload=524174]: Waiting for error group managing segments... Dec 30 13:28:57.954692408 (597h59m57.825775444s) [noise=true , address=145.79.227.46:4603 , piece_id=5FKP2DLPZS4IQIYGKTS7KEGWIJHZMGDDZMOB4XPMDOZIVKEEAFOA, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:57.964731514 (597h59m57.835814559s) [noise=true , address=145.79.227.7:4295 , piece_id=HINKDYRST6Z6QTC7BUYL2NX64UIVIFXZG2HZVUBLQ3YPRZSO47UA, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:57.964823740 (597h59m57.835906775s) [noise=true , address=145.79.227.24:4581 , piece_id=RNQYOP2XOETR3RIO2SXUXWBDPIA7XUAF6PZSQZ6UU3LRVBFWGPCQ, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:57.966627555 (597h59m57.83771061s) [noise=true , address=145.79.227.25:4617 , piece_id=4SX4DV7H5PBGYE2JXQDNJCXJFZHBEMDWMBH623CFM3DGJ2BYWPVQ, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:57.974471009 (597h59m57.845554054s) [noise=true , address=145.79.227.2:4115 , piece_id=7TKKPHCDVVHSUBUAJYA2CDMI4AYGUC5GSGLZVH246KU3GFUZ3IPQ, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:57.976577451 (597h59m57.847660496s) [noise=true , address=145.79.227.45:4231 , piece_id=RGSCYHMMLSATNC7FSB6JPGCJPRPJKBC45WWYEQFLFS7LJR6U2EYQ, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:57.980584150 (597h59m57.851667185s) [noise=true , address=145.79.227.3:4150 , piece_id=PPOHFUDOBD5G4MGNKB3N7CJI64AILUWKR4HVFATHKENWSYJ3UD6A, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:57.982070591 (597h59m57.853153636s) [noise=true , address=145.79.227.26:4653 , piece_id=OIYDLLKF3VKICZHWUPANYXHKZ6UPPJV3LL6IT5KTR6W2YD7LHWCA, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:57.985823918 (597h59m57.856906973s) [noise=true , address=145.79.227.15:4873 , piece_id=QEN7EXHJY22WHEHAS544KWQWUKJ44G4SLHUECPOUAQOESXX2YTKA, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:58.043333133 (597h59m57.914416168s) [noise=true , address=145.79.227.38:4978 , piece_id=2NJCYSCE547YO3QUQLXYRG5YXBCCNSRSADI4NW4AVUAWDA5UWTQA, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:58.048926615 (597h59m57.92000966s) [noise=true , address=145.79.227.8:4331 , piece_id=UTEN5C5VDDGF6X4CLX7P5V336RUGY7AQADV47PMIAI4MVQZWKOQA, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:58.217409791 (597h59m58.088492826s) [noise=true , address=77.90.31.11:4553 , piece_id=NPAJEEBODUMLAPQ6DXSBXN7BQP7A4TEDR3PE52GN34DJ53AYIQ5Q, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:58.687868312 (597h59m58.558951347s) [noise=true , address=45.91.116.8:4046 , piece_id=D36STZV6XH6CHIH2EM5KE66DYJGYDKP6XYSCLTGPVWEANESOM3UQ, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:58.687893490 (597h59m58.558976526s) [seg_pos={0 0}, upload=524174]: Segment reached optimal threshold of 13 pieces. Dec 30 13:28:58.687933927 (597h59m58.559016982s) [noise=true , address=77.90.31.10:4520 , piece_id=K3VJPV7MQXIXB5HDHTZMRKGDM6KDLAUAI2IVFU5TU6Y7LCXEXSFQ, seg_pos={0 0}, upload=524174]: Done uploading piece. err: <nil> Dec 30 13:28:58.688127657 (597h59m58.559210712s) [noise=true , address=145.79.227.44:4195 , piece_id=J4QNUZ7CN44TVECPGHG75PD2TP2P7AUWSSXYR4UULSFCPSDHNJRQ, seg_pos={0 0}, upload=524174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 13:28:58.688142294 (597h59m58.559225339s) [noise=true , address=145.79.227.44:4195 , piece_id=J4QNUZ7CN44TVECPGHG75PD2TP2P7AUWSSXYR4UULSFCPSDHNJRQ, seg_pos={0 0}, upload=524174]: Long tail cancellation due to optimal threshold Dec 30 13:28:58.688370259 (597h59m58.559453294s) [noise=true , address=45.91.116.5:4300 , piece_id=NFBCKO3TO7CENHFR7H4UYTT3R4QKZZA6WMG6BNREUFODODVKK5AA, seg_pos={0 0}, upload=524174]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:38268->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:38268->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38268->45.91.116.5:4300: use of closed network connection Dec 30 13:28:58.688381520 (597h59m58.559464565s) [noise=true , address=45.91.116.5:4300 , piece_id=NFBCKO3TO7CENHFR7H4UYTT3R4QKZZA6WMG6BNREUFODODVKK5AA, seg_pos={0 0}, upload=524174]: Long tail cancellation due to optimal threshold Dec 30 13:28:58.688400937 (597h59m58.559483982s) [seg_pos={0 0}, upload=524174]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 30 13:28:58.688481130 (597h59m58.559564175s) [upload=524174]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:28:58.688487922 (597h59m58.559570977s) [upload=524174]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:28:58.975680930 (597h59m58.846763975s) [upload=524174]: Done uploading segments... Dec 30 13:28:58.975707641 (597h59m58.846790686s) [upload=524174]: Upload finished. err: <nil> Duration: 30.003609738s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X....X................................................] duration: 26.342186611s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:29:08 UTC) orbiter warning: context deadline exceeded Duration: 26.342211088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X....X........X..............X.......X...X..............X.X] duration: 1m0.003542332s trace: https://jaeger.dev.storj.io/trace/215247d8c89b6452 FAIL (2025-12-30 13:29:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.896120295 (597h59m8.76720334s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.896501861 (597h59m8.767584896s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.896884961 (597h59m8.767968006s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.897438555 (597h59m8.7685216s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.897918618 (597h59m8.769001654s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898118489 (597h59m8.769201534s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898293232 (597h59m8.769376267s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898578856 (597h59m8.769661901s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898596830 (597h59m8.769679865s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.898975752 (597h59m8.770058787s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.899325348 (597h59m8.770408383s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.899670295 (597h59m8.77075334s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.899999922 (597h59m8.771082957s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:08.900322136 (597h59m8.771405171s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Uploading piece... Dec 30 13:28:10.351622756 (597h59m10.222705791s) [upload=524159]: Next returned nil segment Dec 30 13:28:10.351638757 (597h59m10.222721792s) [upload=524159]: Waiting for error group managing segments... Dec 30 13:28:10.760041585 (597h59m10.63112463s) [noise=true , address=145.79.227.8:4343 , piece_id=4XVUVKSL6BUFL42NE7S47FLEXAAZ5DJKLJFBCZWCLJP5U4QIFWGA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:10.771328507 (597h59m10.642411542s) [noise=true , address=145.79.227.46:4620 , piece_id=O2LYDU4GUHTIR672MTOXKCIPFBSCDFRFIJVDGOJJVACYPJ6VEZSQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:10.799274151 (597h59m10.670357186s) [noise=true , address=145.79.227.29:4969 , piece_id=BYLTONLWJLW6RSRGJGFCK4R75AYOKYVRTB4LJJVA7PVNRQ2ALQZA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:10.847914048 (597h59m10.718997094s) [noise=true , address=145.79.227.7:4315 , piece_id=FXQECXE32SO35GVQGRIPMSOVU6MQSVNCE7LNYAIUSJKZAVYJY7KQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.084640159 (597h59m10.955723205s) [noise=true , address=77.90.31.19:4847 , piece_id=RQ4HBXVV67X3OT6E5HLJUP7BFHUN3ORS7QHMSHV2KPWG2YOSXE3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.098064000 (597h59m10.969147035s) [noise=true , address=77.90.31.13:4654 , piece_id=5T6IPGFNVQBHEEFR4RPVTY4BWCVNVVAJQ2XHDF2KGLMYDBYA4XYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.137426853 (597h59m11.008509898s) [noise=true , address=77.90.31.18:4836 , piece_id=PXG6VIVR3GKYC7BEWGLGZOTCQRODJHPNACBISUUMUNE74QCBNRAA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.251898794 (597h59m11.122981839s) [noise=true , address=45.91.116.6:4002 , piece_id=2IXI72FZ37L53DLVLYWFHLRJWTKNZWZNAKWMXUWRFF2TORIHWBQQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.596460591 (597h59m11.467543636s) [noise=true , address=77.90.31.17:4788 , piece_id=FTJ53N6JHJMTTVEL2RA536OREDREPEBLET4DQEHW4GVKB6TWK2DQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.639953307 (597h59m11.511036352s) [noise=true , address=145.79.227.30:4768 , piece_id=VAAHYFEUMX2Z2HII35XILZT5MCHECNGRU6F6H2EJAIRDIITLPUVA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.684868994 (597h59m11.555952039s) [noise=true , address=145.79.227.40:4054 , piece_id=ZFFV77Y5QNHXMCDC5UMWCQFPG3PILN5LFBAMA3RS36TS3EY3VD6A, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.754590985 (597h59m11.62567403s) [noise=true , address=77.90.31.10:4549 , piece_id=TD4BHX75IGCPJY5CE3DHT4P7NN6R3NNTC3WFZKQ3QXOVP3FGHKBQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.773404208 (597h59m11.644487243s) [noise=true , address=145.79.227.44:4209 , piece_id=E34RET5D2ON7NM6REIPIAJLT2REIDULAKGGMYEL4FK75L5W5WTYA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.782814085 (597h59m11.65389713s) [noise=true , address=193.149.164.143:4697 , piece_id=V2O2UPXON575Y6DRU55HTW7CGLYP2W24M3CJXPVJH64FRI3BKB3Q, seg_pos={0 0}, upload=524159]: Done uploading piece. err: <nil> Dec 30 13:28:11.782833372 (597h59m11.653916407s) [seg_pos={0 0}, upload=524159]: Segment reached optimal threshold of 14 pieces. Dec 30 13:28:11.783052499 (597h59m11.654135544s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 13:28:11.783058921 (597h59m11.654141966s) [noise=true , address=145.79.227.49:4367 , piece_id=DP23YURVLWGI26FLHJ4HB4CS3KOYO2PPRB64CG5KGMWHFRISAQNA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold Dec 30 13:28:11.783172438 (597h59m11.654255493s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38620->64.49.13.169:4711: use of closed network connection Dec 30 13:28:11.783193367 (597h59m11.654276402s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 30 13:28:11.783202285 (597h59m11.65428532s) [noise=true , address=145.79.227.33:4904 , piece_id=YVOGL6PQ3I5V357RDTKU7ZXZ7ONSQJZWPLAB3ZLTIY3VUUCUZIHA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold Dec 30 13:28:11.783209388 (597h59m11.654292423s) [noise=true , address=64.49.13.169:4711 , piece_id=E7WI5M22YH3YLGYLOQW7QDEXMIYJLUBREZSWVIM4BLOBEZJJ3EKA, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold Dec 30 13:28:11.783339226 (597h59m11.654422271s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36234->145.79.227.35:4511: use of closed network connection Dec 30 13:28:11.783345057 (597h59m11.654428102s) [noise=true , address=145.79.227.35:4511 , piece_id=ZDJ4CKCW42Q4P46QPMXRVH2BWI34VHW2LTYRR3UUO6HA3QRGUQMQ, seg_pos={0 0}, upload=524159]: Long tail cancellation due to optimal threshold Dec 30 13:28:11.783485995 (597h59m11.65456904s) [seg_pos={0 0}, upload=524159]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:28:11.783604620 (597h59m11.654687656s) [upload=524159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:28:11.783615401 (597h59m11.654698466s) [upload=524159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:28:12.079472557 (597h59m11.950555592s) [upload=524159]: Done uploading segments... Dec 30 13:28:12.079493527 (597h59m11.950576562s) [upload=524159]: Upload finished. err: <nil> Duration: 1m0.003560006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXXXX.XXX] duration: 5.00123465s trace: https://jaeger.dev.storj.io/trace/454caae2fb5c354f FAIL (2025-12-30 13:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001252414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X...........X..X........X...X...XXX.......X...XX.XX..] duration: 25.025775358s trace: https://jaeger.dev.storj.io/trace/f9caef817234e2e FAIL (2025-12-30 13:28:25 UTC) orbiter warning: context deadline exceeded Duration: 25.025801367s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration: 21.003829096s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:28:20 UTC) orbiter warning: context deadline exceeded Duration: 21.003839987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X..X.X..X....X........XX................X.XX....XX..X] duration: 30.00323683s trace: https://jaeger.dev.storj.io/trace/57231434d6377687 FAIL (2025-12-30 13:28:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: HEI4LOX6JFNKM2XYU7ZSFTCDOGRLWI7VLQ6SV523JDTKU7J3ATPA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: EKJVHLUEBHCGRHUAPEC2PCNHAIDI7MQEM37TGLS7DYRTDTDPFUOQ) context deadline exceeded ======LOGS====== Dec 30 13:27:47.666903925 (597h58m47.53798696s) [upload=524156]: Starting upload Dec 30 13:27:47.666955993 (597h58m47.538039038s) [upload=524156]: Done starting upload Dec 30 13:27:47.667050243 (597h58m47.538133308s) [upload=524156]: Uploading segments... Dec 30 13:27:47.667148330 (597h58m47.538231365s) [upload=524156]: Got next segment. Inline: false Dec 30 13:27:47.667180611 (597h58m47.538263646s) [upload=524156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:27:47.667184900 (597h58m47.538267935s) [upload=524156]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:27:47.918153829 (597h58m47.789236864s) [seg_pos={0 0}, upload=524156]: Begin upload segment... Dec 30 13:27:47.918198435 (597h58m47.78928147s) [seg_pos={0 0}, upload=524156]: Done begin upload segment. Dec 30 13:27:47.918214946 (597h58m47.789297981s) [noise=true , address=145.79.227.3:4149 , piece_id=T4I3BMT7YS3IMKIDEE7YJMPYVHWYCGQM2MI54O2F3KSJAESZF2IA, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.918224153 (597h58m47.789307188s) [noise=true , address=145.79.227.25:4617 , piece_id=ERITIJDFKXYL5TA2J3QV4ZOFKKOPPJBRMOEZPOQRAWPUVIOHJROA, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.918680682 (597h58m47.789763717s) [noise=true , address=77.90.31.19:4844 , piece_id=HEI4LOX6JFNKM2XYU7ZSFTCDOGRLWI7VLQ6SV523JDTKU7J3ATPA, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.919051539 (597h58m47.790134574s) [noise=true , address=145.79.227.16:4910 , piece_id=QGJZKYQS7EAPO63RKNXTYPIAABLM75XDROXCHZBD7I5UYVZAEM5Q, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.919338766 (597h58m47.790421811s) [noise=true , address=145.79.227.4:4186 , piece_id=N4FB6SQR2INCL7AW4YXI5JYAENNYLGBIFC2AVQZ2YYO7MINF3NBQ, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.919595134 (597h58m47.790678169s) [noise=true , address=145.79.227.14:4840 , piece_id=URQLL27FGDEALKQEHNR5WW4C6Z7D54GPN54HAYCRV2GGSQFIOROA, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.919908069 (597h58m47.790991105s) [noise=true , address=145.79.227.28:4726 , piece_id=ZRQLMSXHGXD7O4XVI4JRUIC673WCZKOEQCS7PONITQZ2HSQ3FVZA, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.920200527 (597h58m47.791283572s) [noise=true , address=77.90.31.14:4662 , piece_id=EKJVHLUEBHCGRHUAPEC2PCNHAIDI7MQEM37TGLS7DYRTDTDPFUOQ, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.920499115 (597h58m47.79158216s) [noise=true , address=145.79.227.52:4447 , piece_id=GTPZ5QVPKU2GITBFWDUX6WN7EKDNXXQ6YITCCRF3SMZKGOUCPCFQ, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.922430824 (597h58m47.793513859s) [noise=true , address=145.79.227.10:4403 , piece_id=5LBXPGH7IS2QGX6QDTBCC2OVJCVKTBHWDEUG7HVHPKL2JFPZPCFA, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.923692015 (597h58m47.79477506s) [noise=true , address=45.91.116.5:4299 , piece_id=WPDGY6K6YJBRWBDVVXRIJ3FXKR7SFFNKBE7SFZY755XK46SG4OCQ, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.924710885 (597h58m47.79579395s) [noise=true , address=145.79.227.37:4566 , piece_id=HPNDZP4QPPGSGGBM3CU64RU263MFVFHMRH557RSF4Y4MPWOHPVJA, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.925843251 (597h58m47.796926286s) [noise=true , address=145.79.227.21:4473 , piece_id=UWGXOVL7T2EIA5V2CSGSI7VFCWP74IC26EGAPWNQRBMLUDLUI65Q, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.927776692 (597h58m47.798859727s) [noise=true , address=77.90.31.10:4518 , piece_id=HULFQQBHBXRAVEPAJD4QHJMGOYAQJWWQS35CTA2AXSS3ADN3WUBA, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.927878536 (597h58m47.798961571s) [noise=true , address=145.79.227.13:4804 , piece_id=FFOYWROK7GKVTIZH7OE5E65TJBKY2GZH4CDBXGP5MQ7KBJN3QP7A, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:47.928936602 (597h58m47.800019637s) [noise=true , address=145.79.227.46:4602 , piece_id=IMNVQFGDXIJYLM5D6S2BNNE5NCMGB33JVZO4N2U6V76FQJDH2P2A, seg_pos={0 0}, upload=524156]: Uploading piece... Dec 30 13:27:49.628439024 (597h58m49.499522069s) [upload=524156]: Next returned nil segment Dec 30 13:27:49.628451958 (597h58m49.499535003s) [upload=524156]: Waiting for error group managing segments... Dec 30 13:27:49.902415145 (597h58m49.77349818s) [noise=true , address=145.79.227.14:4840 , piece_id=URQLL27FGDEALKQEHNR5WW4C6Z7D54GPN54HAYCRV2GGSQFIOROA, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.909276149 (597h58m49.780359194s) [noise=true , address=145.79.227.25:4617 , piece_id=ERITIJDFKXYL5TA2J3QV4ZOFKKOPPJBRMOEZPOQRAWPUVIOHJROA, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.910869112 (597h58m49.781952147s) [noise=true , address=145.79.227.52:4447 , piece_id=GTPZ5QVPKU2GITBFWDUX6WN7EKDNXXQ6YITCCRF3SMZKGOUCPCFQ, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.915693277 (597h58m49.786776323s) [noise=true , address=145.79.227.13:4804 , piece_id=FFOYWROK7GKVTIZH7OE5E65TJBKY2GZH4CDBXGP5MQ7KBJN3QP7A, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.915771407 (597h58m49.786854442s) [noise=true , address=145.79.227.4:4186 , piece_id=N4FB6SQR2INCL7AW4YXI5JYAENNYLGBIFC2AVQZ2YYO7MINF3NBQ, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.921251872 (597h58m49.792334907s) [noise=true , address=145.79.227.21:4473 , piece_id=UWGXOVL7T2EIA5V2CSGSI7VFCWP74IC26EGAPWNQRBMLUDLUI65Q, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.931409745 (597h58m49.80249278s) [noise=true , address=145.79.227.3:4149 , piece_id=T4I3BMT7YS3IMKIDEE7YJMPYVHWYCGQM2MI54O2F3KSJAESZF2IA, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.943079475 (597h58m49.81416251s) [noise=true , address=145.79.227.37:4566 , piece_id=HPNDZP4QPPGSGGBM3CU64RU263MFVFHMRH557RSF4Y4MPWOHPVJA, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.997739866 (597h58m49.868822901s) [noise=true , address=145.79.227.46:4602 , piece_id=IMNVQFGDXIJYLM5D6S2BNNE5NCMGB33JVZO4N2U6V76FQJDH2P2A, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:49.998328878 (597h58m49.869411923s) [noise=true , address=145.79.227.28:4726 , piece_id=ZRQLMSXHGXD7O4XVI4JRUIC673WCZKOEQCS7PONITQZ2HSQ3FVZA, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:50.001450723 (597h58m49.872533768s) [noise=true , address=145.79.227.10:4403 , piece_id=5LBXPGH7IS2QGX6QDTBCC2OVJCVKTBHWDEUG7HVHPKL2JFPZPCFA, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:50.098123813 (597h58m49.969206858s) [noise=true , address=77.90.31.19:4844 , piece_id=HEI4LOX6JFNKM2XYU7ZSFTCDOGRLWI7VLQ6SV523JDTKU7J3ATPA, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:50.138304633 (597h58m50.009387678s) [noise=true , address=77.90.31.14:4662 , piece_id=EKJVHLUEBHCGRHUAPEC2PCNHAIDI7MQEM37TGLS7DYRTDTDPFUOQ, seg_pos={0 0}, upload=524156]: Done uploading piece. err: <nil> Dec 30 13:27:50.138330302 (597h58m50.009413357s) [seg_pos={0 0}, upload=524156]: Segment reached optimal threshold of 13 pieces. Dec 30 13:27:50.138588423 (597h58m50.009671468s) [noise=true , address=45.91.116.5:4299 , piece_id=WPDGY6K6YJBRWBDVVXRIJ3FXKR7SFFNKBE7SFZY755XK46SG4OCQ, seg_pos={0 0}, upload=524156]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 13:27:50.138599124 (597h58m50.009682169s) [noise=true , address=45.91.116.5:4299 , piece_id=WPDGY6K6YJBRWBDVVXRIJ3FXKR7SFFNKBE7SFZY755XK46SG4OCQ, seg_pos={0 0}, upload=524156]: Long tail cancellation due to optimal threshold Dec 30 13:27:50.138741445 (597h58m50.00982449s) [noise=true , address=145.79.227.16:4910 , piece_id=QGJZKYQS7EAPO63RKNXTYPIAABLM75XDROXCHZBD7I5UYVZAEM5Q, seg_pos={0 0}, upload=524156]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 30 13:27:50.138760010 (597h58m50.009843045s) [noise=true , address=145.79.227.16:4910 , piece_id=QGJZKYQS7EAPO63RKNXTYPIAABLM75XDROXCHZBD7I5UYVZAEM5Q, seg_pos={0 0}, upload=524156]: Long tail cancellation due to optimal threshold Dec 30 13:27:50.138912120 (597h58m50.009995155s) [noise=true , address=77.90.31.10:4518 , piece_id=HULFQQBHBXRAVEPAJD4QHJMGOYAQJWWQS35CTA2AXSS3ADN3WUBA, seg_pos={0 0}, upload=524156]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:45500->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:45500->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->77.90.31.10:4518: use of closed network connection Dec 30 13:27:50.138921919 (597h58m50.010004974s) [noise=true , address=77.90.31.10:4518 , piece_id=HULFQQBHBXRAVEPAJD4QHJMGOYAQJWWQS35CTA2AXSS3ADN3WUBA, seg_pos={0 0}, upload=524156]: Long tail cancellation due to optimal threshold Dec 30 13:27:50.138951726 (597h58m50.010034761s) [seg_pos={0 0}, upload=524156]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:27:50.139041696 (597h58m50.010124741s) [upload=524156]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:27:50.139048840 (597h58m50.010131885s) [upload=524156]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:27:50.433976787 (597h58m50.305059822s) [upload=524156]: Done uploading segments... Dec 30 13:27:50.433995562 (597h58m50.305078607s) [upload=524156]: Upload finished. err: <nil> Duration: 30.003258612s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X....X..............................................] duration: 36.622441737s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:27:59 UTC) orbiter warning: context deadline exceeded Duration: 36.622469452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X...........X..X........X...X...XXX.......X...XX.XX.] duration: 21.318562889s trace: https://jaeger.dev.storj.io/trace/f9caef817234e2e FAIL (2025-12-30 13:27:39 UTC) orbiter warning: context deadline exceeded Duration: 21.318577296s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXXXX.XX] duration: 5.00154711s trace: https://jaeger.dev.storj.io/trace/79b0736517de0da1 FAIL (2025-12-30 13:27:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00156261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....X........X..............X.......X...X..............X.] duration: 15.445338535s trace: https://jaeger.dev.storj.io/trace/6007e435b9dd3942 FAIL (2025-12-30 13:27:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, Piece ID: 27X3HSVW6Q56P7D2T3M7YPHG7WJQGX26UH5Y6S2CTBGEXCODP3QQ) context deadline exceeded; piecestore: (Node ID: 12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, Piece ID: WXTVMIFJD2ZKKWAYW677HQJGO6G5HJET5DCFNPJJ3PGU3VZBK7SA) context deadline exceeded; piecestore: (Node ID: 12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, Piece ID: EO4VCHVZ6TZDWL6VGIRBNDE4OIXBXS5DHUYKKNJCDSA33BDG73DA) context deadline exceeded ======LOGS====== m8.861650408s) [noise=true , address=64.49.13.169:4716 , piece_id=5NTZ3U4KSZ2ZBVGUKHD5RIONNI67OOBU6SNRA6NIHVM726GWGUOQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.991305839 (597h57m8.862388884s) [noise=true , address=145.79.227.40:4057 , piece_id=IZGJXM5PKQDQ3UFBXC6GIZGOZYATOSJDPO7MUB4AHV42WNS46APQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.991819768 (597h57m8.862902813s) [noise=true , address=145.79.227.33:4882 , piece_id=4IRKGINK2GWO5ZBX6MI4WEQKNIFRV53PK7BXM5Q4QLG6BC5LFW2A, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.991953182 (597h57m8.863036227s) [noise=true , address=145.79.227.30:4796 , piece_id=HGUUYCEUID4RB55HB2LQ4DV63V3UHCE4KADFWVXVB7TZQVH7ZERA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992151189 (597h57m8.863234224s) [noise=true , address=145.79.227.44:4220 , piece_id=U3RH6Q4BULNRXGEINXFR3WOOEPLTWQPBQJHNJIWTFTSIVZKQY73Q, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992306375 (597h57m8.86338941s) [noise=true , address=145.79.227.18:4009 , piece_id=NKNAAXYYE3FJE7H2BHQC2RZFZ7E34I2JW6CIFHC752BNSSJJBEZA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992502057 (597h57m8.863585102s) [noise=true , address=145.79.227.34:4909 , piece_id=MFD46VNSFDSFPPO5TREMGVQVBMYBGLGWBRGE4LQ2BU2LHBGZ4CCQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992795867 (597h57m8.863878902s) [noise=true , address=193.149.164.143:4700 , piece_id=LZF5UGZFZSUZSC52J7DSGXV5MDGXI7CZS2ZHCPF3IRTD7KS52KMQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992850079 (597h57m8.863933115s) [noise=true , address=145.79.227.29:4964 , piece_id=YMR32X6T57ADU5TVFU4OATYO5FVSTMWITMILHUI6LQ6BDXKOWRJA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993113081 (597h57m8.864196116s) [noise=true , address=45.91.116.6:4000 , piece_id=27X3HSVW6Q56P7D2T3M7YPHG7WJQGX26UH5Y6S2CTBGEXCODP3QQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993124542 (597h57m8.864207587s) [noise=true , address=145.79.227.43:4187 , piece_id=P7DD5RI3M2UFJKPKBAD4DRVSIZOIFH5OAZOOEKLGIY3IK72JWA4Q, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993369579 (597h57m8.864452614s) [noise=true , address=77.90.31.16:4751 , piece_id=EO4VCHVZ6TZDWL6VGIRBNDE4OIXBXS5DHUYKKNJCDSA33BDG73DA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993412331 (597h57m8.864495366s) [noise=true , address=45.91.116.8:4052 , piece_id=6VMOZQ4RQAJT7D7SYGVGUCPNDART2JJI7ACXBNHXIO33HJVXMZ2A, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993740305 (597h57m8.86482334s) [noise=true , address=64.49.13.154:4931 , piece_id=HSRSG2WXV6PM52AZWHV36TWRR67APTLC4KGBMLEF7KMQXTV26U4A, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.994015629 (597h57m8.865098674s) [noise=true , address=77.90.31.21:4910 , piece_id=WXTVMIFJD2ZKKWAYW677HQJGO6G5HJET5DCFNPJJ3PGU3VZBK7SA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.994044545 (597h57m8.86512758s) [noise=true , address=145.79.227.49:4356 , piece_id=5LSNXG6IMDXPN6G6WLK6RWO7LJAIZCVR4ZLVIK6ZIS356S6XJVEQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:10.438067227 (597h57m10.309150272s) [upload=524131]: Next returned nil segment Dec 30 13:26:10.438086633 (597h57m10.309169669s) [upload=524131]: Waiting for error group managing segments... Dec 30 13:26:10.829678612 (597h57m10.700761657s) [noise=true , address=64.49.13.154:4931 , piece_id=HSRSG2WXV6PM52AZWHV36TWRR67APTLC4KGBMLEF7KMQXTV26U4A, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:10.865732003 (597h57m10.736815038s) [noise=true , address=145.79.227.43:4187 , piece_id=P7DD5RI3M2UFJKPKBAD4DRVSIZOIFH5OAZOOEKLGIY3IK72JWA4Q, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:10.932185341 (597h57m10.803268376s) [noise=true , address=145.79.227.29:4964 , piece_id=YMR32X6T57ADU5TVFU4OATYO5FVSTMWITMILHUI6LQ6BDXKOWRJA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:10.948118610 (597h57m10.819201645s) [noise=true , address=145.79.227.18:4009 , piece_id=NKNAAXYYE3FJE7H2BHQC2RZFZ7E34I2JW6CIFHC752BNSSJJBEZA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.158150181 (597h57m11.029233226s) [noise=true , address=77.90.31.16:4751 , piece_id=EO4VCHVZ6TZDWL6VGIRBNDE4OIXBXS5DHUYKKNJCDSA33BDG73DA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.199855213 (597h57m11.070938248s) [noise=true , address=77.90.31.21:4910 , piece_id=WXTVMIFJD2ZKKWAYW677HQJGO6G5HJET5DCFNPJJ3PGU3VZBK7SA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.266669548 (597h57m11.137752583s) [noise=true , address=45.91.116.8:4052 , piece_id=6VMOZQ4RQAJT7D7SYGVGUCPNDART2JJI7ACXBNHXIO33HJVXMZ2A, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.673401184 (597h57m11.544484219s) [noise=true , address=45.91.116.6:4000 , piece_id=27X3HSVW6Q56P7D2T3M7YPHG7WJQGX26UH5Y6S2CTBGEXCODP3QQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.729771361 (597h57m11.600854396s) [noise=true , address=145.79.227.44:4220 , piece_id=U3RH6Q4BULNRXGEINXFR3WOOEPLTWQPBQJHNJIWTFTSIVZKQY73Q, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.733292948 (597h57m11.604375983s) [noise=true , address=77.90.31.18:4813 , piece_id=XMG3IPFIV3FDM6D44CRF6XHUFGB2Q2RJG2IZDGCFOXBWZSLXEHOA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.764294869 (597h57m11.635377904s) [noise=true , address=145.79.227.30:4796 , piece_id=HGUUYCEUID4RB55HB2LQ4DV63V3UHCE4KADFWVXVB7TZQVH7ZERA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.829405059 (597h57m11.700488104s) [noise=true , address=145.79.227.40:4057 , piece_id=IZGJXM5PKQDQ3UFBXC6GIZGOZYATOSJDPO7MUB4AHV42WNS46APQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.844577138 (597h57m11.715660183s) [noise=true , address=193.149.164.143:4700 , piece_id=LZF5UGZFZSUZSC52J7DSGXV5MDGXI7CZS2ZHCPF3IRTD7KS52KMQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.899218584 (597h57m11.770301629s) [noise=true , address=145.79.227.33:4882 , piece_id=4IRKGINK2GWO5ZBX6MI4WEQKNIFRV53PK7BXM5Q4QLG6BC5LFW2A, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.899250876 (597h57m11.770333911s) [seg_pos={0 0}, upload=524131]: Segment reached optimal threshold of 14 pieces. Dec 30 13:26:11.899492757 (597h57m11.770575792s) [noise=true , address=77.90.31.19:4845 , piece_id=2VVT77OPOK54NFEDEWTVJEJZDKNJKVND6CQX7FDR43SHZYKSGTYQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Dec 30 13:26:11.899499720 (597h57m11.770582755s) [noise=true , address=77.90.31.19:4845 , piece_id=2VVT77OPOK54NFEDEWTVJEJZDKNJKVND6CQX7FDR43SHZYKSGTYQ, seg_pos={0 0}, upload=524131]: Long tail cancellation due to optimal threshold Dec 30 13:26:11.899742422 (597h57m11.770825467s) [noise=true , address=145.79.227.34:4909 , piece_id=MFD46VNSFDSFPPO5TREMGVQVBMYBGLGWBRGE4LQ2BU2LHBGZ4CCQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 30 13:26:11.899780785 (597h57m11.77086382s) [noise=true , address=145.79.227.34:4909 , piece_id=MFD46VNSFDSFPPO5TREMGVQVBMYBGLGWBRGE4LQ2BU2LHBGZ4CCQ, seg_pos={0 0}, upload=524131]: Long tail cancellation due to optimal threshold Dec 30 13:26:11.899798468 (597h57m11.770881613s) [noise=true , address=145.79.227.49:4356 , piece_id=5LSNXG6IMDXPN6G6WLK6RWO7LJAIZCVR4ZLVIK6ZIS356S6XJVEQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: expected piece hash; context canceled; EOF Dec 30 13:26:11.899815842 (597h57m11.770898877s) [noise=true , address=145.79.227.49:4356 , piece_id=5LSNXG6IMDXPN6G6WLK6RWO7LJAIZCVR4ZLVIK6ZIS356S6XJVEQ, seg_pos={0 0}, upload=524131]: Long tail cancellation due to optimal threshold Dec 30 13:26:11.899906584 (597h57m11.770989619s) [noise=true , address=64.49.13.169:4716 , piece_id=5NTZ3U4KSZ2ZBVGUKHD5RIONNI67OOBU6SNRA6NIHVM726GWGUOQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:57434->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:57434->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57434->64.49.13.169:4716: use of closed network connection Dec 30 13:26:11.899914249 (597h57m11.770997284s) [noise=true , address=64.49.13.169:4716 , piece_id=5NTZ3U4KSZ2ZBVGUKHD5RIONNI67OOBU6SNRA6NIHVM726GWGUOQ, seg_pos={0 0}, upload=524131]: Long tail cancellation due to optimal threshold Dec 30 13:26:11.899932503 (597h57m11.771015538s) [seg_pos={0 0}, upload=524131]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:26:11.899999731 (597h57m11.771082776s) [upload=524131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:26:11.900005081 (597h57m11.771088116s) [upload=524131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:26:12.188037167 (597h57m12.059120202s) [upload=524131]: Done uploading segments... Dec 30 13:26:12.188059089 (597h57m12.059142124s) [upload=524131]: Upload finished. err: <nil> Duration: 15.445358543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X........X...........X..X........X...X...XXX.......X...XX.XX] duration: 30.001256381s trace: https://jaeger.dev.storj.io/trace/f9caef817234e2e FAIL (2025-12-30 13:27:17 UTC) orbiter warning: context deadline exceeded Duration: 30.001275718s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....X........X..............X.......X...X..............X] duration: 1m0.00537071s trace: https://jaeger.dev.storj.io/trace/6007e435b9dd3942 FAIL (2025-12-30 13:27:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, Piece ID: 27X3HSVW6Q56P7D2T3M7YPHG7WJQGX26UH5Y6S2CTBGEXCODP3QQ) context deadline exceeded; piecestore: (Node ID: 12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, Piece ID: WXTVMIFJD2ZKKWAYW677HQJGO6G5HJET5DCFNPJJ3PGU3VZBK7SA) context deadline exceeded; piecestore: (Node ID: 12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, Piece ID: EO4VCHVZ6TZDWL6VGIRBNDE4OIXBXS5DHUYKKNJCDSA33BDG73DA) context deadline exceeded ======LOGS====== m8.861650408s) [noise=true , address=64.49.13.169:4716 , piece_id=5NTZ3U4KSZ2ZBVGUKHD5RIONNI67OOBU6SNRA6NIHVM726GWGUOQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.991305839 (597h57m8.862388884s) [noise=true , address=145.79.227.40:4057 , piece_id=IZGJXM5PKQDQ3UFBXC6GIZGOZYATOSJDPO7MUB4AHV42WNS46APQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.991819768 (597h57m8.862902813s) [noise=true , address=145.79.227.33:4882 , piece_id=4IRKGINK2GWO5ZBX6MI4WEQKNIFRV53PK7BXM5Q4QLG6BC5LFW2A, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.991953182 (597h57m8.863036227s) [noise=true , address=145.79.227.30:4796 , piece_id=HGUUYCEUID4RB55HB2LQ4DV63V3UHCE4KADFWVXVB7TZQVH7ZERA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992151189 (597h57m8.863234224s) [noise=true , address=145.79.227.44:4220 , piece_id=U3RH6Q4BULNRXGEINXFR3WOOEPLTWQPBQJHNJIWTFTSIVZKQY73Q, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992306375 (597h57m8.86338941s) [noise=true , address=145.79.227.18:4009 , piece_id=NKNAAXYYE3FJE7H2BHQC2RZFZ7E34I2JW6CIFHC752BNSSJJBEZA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992502057 (597h57m8.863585102s) [noise=true , address=145.79.227.34:4909 , piece_id=MFD46VNSFDSFPPO5TREMGVQVBMYBGLGWBRGE4LQ2BU2LHBGZ4CCQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992795867 (597h57m8.863878902s) [noise=true , address=193.149.164.143:4700 , piece_id=LZF5UGZFZSUZSC52J7DSGXV5MDGXI7CZS2ZHCPF3IRTD7KS52KMQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.992850079 (597h57m8.863933115s) [noise=true , address=145.79.227.29:4964 , piece_id=YMR32X6T57ADU5TVFU4OATYO5FVSTMWITMILHUI6LQ6BDXKOWRJA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993113081 (597h57m8.864196116s) [noise=true , address=45.91.116.6:4000 , piece_id=27X3HSVW6Q56P7D2T3M7YPHG7WJQGX26UH5Y6S2CTBGEXCODP3QQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993124542 (597h57m8.864207587s) [noise=true , address=145.79.227.43:4187 , piece_id=P7DD5RI3M2UFJKPKBAD4DRVSIZOIFH5OAZOOEKLGIY3IK72JWA4Q, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993369579 (597h57m8.864452614s) [noise=true , address=77.90.31.16:4751 , piece_id=EO4VCHVZ6TZDWL6VGIRBNDE4OIXBXS5DHUYKKNJCDSA33BDG73DA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993412331 (597h57m8.864495366s) [noise=true , address=45.91.116.8:4052 , piece_id=6VMOZQ4RQAJT7D7SYGVGUCPNDART2JJI7ACXBNHXIO33HJVXMZ2A, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.993740305 (597h57m8.86482334s) [noise=true , address=64.49.13.154:4931 , piece_id=HSRSG2WXV6PM52AZWHV36TWRR67APTLC4KGBMLEF7KMQXTV26U4A, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.994015629 (597h57m8.865098674s) [noise=true , address=77.90.31.21:4910 , piece_id=WXTVMIFJD2ZKKWAYW677HQJGO6G5HJET5DCFNPJJ3PGU3VZBK7SA, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:08.994044545 (597h57m8.86512758s) [noise=true , address=145.79.227.49:4356 , piece_id=5LSNXG6IMDXPN6G6WLK6RWO7LJAIZCVR4ZLVIK6ZIS356S6XJVEQ, seg_pos={0 0}, upload=524131]: Uploading piece... Dec 30 13:26:10.438067227 (597h57m10.309150272s) [upload=524131]: Next returned nil segment Dec 30 13:26:10.438086633 (597h57m10.309169669s) [upload=524131]: Waiting for error group managing segments... Dec 30 13:26:10.829678612 (597h57m10.700761657s) [noise=true , address=64.49.13.154:4931 , piece_id=HSRSG2WXV6PM52AZWHV36TWRR67APTLC4KGBMLEF7KMQXTV26U4A, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:10.865732003 (597h57m10.736815038s) [noise=true , address=145.79.227.43:4187 , piece_id=P7DD5RI3M2UFJKPKBAD4DRVSIZOIFH5OAZOOEKLGIY3IK72JWA4Q, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:10.932185341 (597h57m10.803268376s) [noise=true , address=145.79.227.29:4964 , piece_id=YMR32X6T57ADU5TVFU4OATYO5FVSTMWITMILHUI6LQ6BDXKOWRJA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:10.948118610 (597h57m10.819201645s) [noise=true , address=145.79.227.18:4009 , piece_id=NKNAAXYYE3FJE7H2BHQC2RZFZ7E34I2JW6CIFHC752BNSSJJBEZA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.158150181 (597h57m11.029233226s) [noise=true , address=77.90.31.16:4751 , piece_id=EO4VCHVZ6TZDWL6VGIRBNDE4OIXBXS5DHUYKKNJCDSA33BDG73DA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.199855213 (597h57m11.070938248s) [noise=true , address=77.90.31.21:4910 , piece_id=WXTVMIFJD2ZKKWAYW677HQJGO6G5HJET5DCFNPJJ3PGU3VZBK7SA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.266669548 (597h57m11.137752583s) [noise=true , address=45.91.116.8:4052 , piece_id=6VMOZQ4RQAJT7D7SYGVGUCPNDART2JJI7ACXBNHXIO33HJVXMZ2A, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.673401184 (597h57m11.544484219s) [noise=true , address=45.91.116.6:4000 , piece_id=27X3HSVW6Q56P7D2T3M7YPHG7WJQGX26UH5Y6S2CTBGEXCODP3QQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.729771361 (597h57m11.600854396s) [noise=true , address=145.79.227.44:4220 , piece_id=U3RH6Q4BULNRXGEINXFR3WOOEPLTWQPBQJHNJIWTFTSIVZKQY73Q, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.733292948 (597h57m11.604375983s) [noise=true , address=77.90.31.18:4813 , piece_id=XMG3IPFIV3FDM6D44CRF6XHUFGB2Q2RJG2IZDGCFOXBWZSLXEHOA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.764294869 (597h57m11.635377904s) [noise=true , address=145.79.227.30:4796 , piece_id=HGUUYCEUID4RB55HB2LQ4DV63V3UHCE4KADFWVXVB7TZQVH7ZERA, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.829405059 (597h57m11.700488104s) [noise=true , address=145.79.227.40:4057 , piece_id=IZGJXM5PKQDQ3UFBXC6GIZGOZYATOSJDPO7MUB4AHV42WNS46APQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.844577138 (597h57m11.715660183s) [noise=true , address=193.149.164.143:4700 , piece_id=LZF5UGZFZSUZSC52J7DSGXV5MDGXI7CZS2ZHCPF3IRTD7KS52KMQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.899218584 (597h57m11.770301629s) [noise=true , address=145.79.227.33:4882 , piece_id=4IRKGINK2GWO5ZBX6MI4WEQKNIFRV53PK7BXM5Q4QLG6BC5LFW2A, seg_pos={0 0}, upload=524131]: Done uploading piece. err: <nil> Dec 30 13:26:11.899250876 (597h57m11.770333911s) [seg_pos={0 0}, upload=524131]: Segment reached optimal threshold of 14 pieces. Dec 30 13:26:11.899492757 (597h57m11.770575792s) [noise=true , address=77.90.31.19:4845 , piece_id=2VVT77OPOK54NFEDEWTVJEJZDKNJKVND6CQX7FDR43SHZYKSGTYQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Dec 30 13:26:11.899499720 (597h57m11.770582755s) [noise=true , address=77.90.31.19:4845 , piece_id=2VVT77OPOK54NFEDEWTVJEJZDKNJKVND6CQX7FDR43SHZYKSGTYQ, seg_pos={0 0}, upload=524131]: Long tail cancellation due to optimal threshold Dec 30 13:26:11.899742422 (597h57m11.770825467s) [noise=true , address=145.79.227.34:4909 , piece_id=MFD46VNSFDSFPPO5TREMGVQVBMYBGLGWBRGE4LQ2BU2LHBGZ4CCQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 30 13:26:11.899780785 (597h57m11.77086382s) [noise=true , address=145.79.227.34:4909 , piece_id=MFD46VNSFDSFPPO5TREMGVQVBMYBGLGWBRGE4LQ2BU2LHBGZ4CCQ, seg_pos={0 0}, upload=524131]: Long tail cancellation due to optimal threshold Dec 30 13:26:11.899798468 (597h57m11.770881613s) [noise=true , address=145.79.227.49:4356 , piece_id=5LSNXG6IMDXPN6G6WLK6RWO7LJAIZCVR4ZLVIK6ZIS356S6XJVEQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: expected piece hash; context canceled; EOF Dec 30 13:26:11.899815842 (597h57m11.770898877s) [noise=true , address=145.79.227.49:4356 , piece_id=5LSNXG6IMDXPN6G6WLK6RWO7LJAIZCVR4ZLVIK6ZIS356S6XJVEQ, seg_pos={0 0}, upload=524131]: Long tail cancellation due to optimal threshold Dec 30 13:26:11.899906584 (597h57m11.770989619s) [noise=true , address=64.49.13.169:4716 , piece_id=5NTZ3U4KSZ2ZBVGUKHD5RIONNI67OOBU6SNRA6NIHVM726GWGUOQ, seg_pos={0 0}, upload=524131]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:57434->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:57434->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57434->64.49.13.169:4716: use of closed network connection Dec 30 13:26:11.899914249 (597h57m11.770997284s) [noise=true , address=64.49.13.169:4716 , piece_id=5NTZ3U4KSZ2ZBVGUKHD5RIONNI67OOBU6SNRA6NIHVM726GWGUOQ, seg_pos={0 0}, upload=524131]: Long tail cancellation due to optimal threshold Dec 30 13:26:11.899932503 (597h57m11.771015538s) [seg_pos={0 0}, upload=524131]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:26:11.899999731 (597h57m11.771082776s) [upload=524131]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:26:11.900005081 (597h57m11.771088116s) [upload=524131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:26:12.188037167 (597h57m12.059120202s) [upload=524131]: Done uploading segments... Dec 30 13:26:12.188059089 (597h57m12.059142124s) [upload=524131]: Upload finished. err: <nil> Duration: 1m0.005384066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXXXX.X] duration: 5.001082819s trace: https://jaeger.dev.storj.io/trace/de140ebbfb3025f FAIL (2025-12-30 13:27:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001107948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X..X.X..X....X........XX................X.XX....XX..] duration: 8.177560985s trace: https://jaeger.dev.storj.io/trace/3f971db36ddeca24 FAIL (2025-12-30 13:27:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 13:25:38.079780045 (597h56m37.95086308s) [upload=524125]: Starting upload Dec 30 13:25:38.079836973 (597h56m37.950920008s) [upload=524125]: Done starting upload Dec 30 13:25:38.080027135 (597h56m37.95111017s) [upload=524125]: Uploading segments... Dec 30 13:25:38.080066340 (597h56m37.951149376s) [upload=524125]: Got next segment. Inline: false Dec 30 13:25:38.080128628 (597h56m37.951211663s) [upload=524125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:25:38.080134641 (597h56m37.951217676s) [upload=524125]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:25:38.329259962 (597h56m38.200343017s) [seg_pos={0 0}, upload=524125]: Begin upload segment... Dec 30 13:25:38.329319876 (597h56m38.200402921s) [seg_pos={0 0}, upload=524125]: Done begin upload segment. Dec 30 13:25:38.329348531 (597h56m38.200431566s) [noise=true , address=145.79.227.47:4265 , piece_id=6GQJA2CDH234RH3RA7Y4D643SUNLT5K44PFHNZ4UABMLPJVWKKKQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.329365673 (597h56m38.200448708s) [noise=true , address=145.79.227.45:4229 , piece_id=QOB22CQKW5NP2ZKY35JT7QFA4ZVQG7GRGDWXXJFCYSRQ55K7IKQQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.329842341 (597h56m38.200925376s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330208908 (597h56m38.201291943s) [noise=true , address=45.91.116.12:4226 , piece_id=NP2HBC7JM233MRXD4AEMGEHVPCJS45JH36DRI5F3JQSZFEINJZOA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330396787 (597h56m38.201479822s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330724059 (597h56m38.201807095s) [noise=true , address=145.79.227.13:4804 , piece_id=F23E2YSHHBJ3F6Y5VGNKXZYGSKK6KVV3TDBCDT5DGG5SI6J5YVVA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331058716 (597h56m38.202141761s) [noise=true , address=77.90.31.14:4664 , piece_id=TBG2LWSXBYVLOO456JJDS34O6CI5APU2J2JBONCGY2BQ72TWVUSQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331433981 (597h56m38.202517016s) [noise=true , address=145.79.227.3:4149 , piece_id=7IQ5GONSR2A2AHH6TIWD2EIHAYFM3FV5VQJBHBZ7W54NFZYBHUXQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331567595 (597h56m38.202650631s) [noise=true , address=145.79.227.12:4765 , piece_id=JI3TAHCJKJBVASYTI5IGZPF7EWWDTORGYJOER4NNNDGGS3JZHG6Q, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331754090 (597h56m38.202837125s) [noise=true , address=145.79.227.19:4079 , piece_id=TPPGIPDU2MHYFH4U57PN5PWRW767NKA2OLZVPMPLCDGG4GIKXOQQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331958239 (597h56m38.203041274s) [noise=true , address=145.79.227.29:4943 , piece_id=MLF4KPYLI5GXMN76RJYFC24NU5J3GTDJZFEO3NYAMT5MVCMY4TOA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332059652 (597h56m38.203142687s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332380894 (597h56m38.203463929s) [noise=true , address=145.79.227.52:4447 , piece_id=O7H7SBH24ADRRGIM6I5ZTAIO57SRL3MRBLDAOXJ6TUJIRKSVLEDA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332540628 (597h56m38.203623663s) [noise=true , address=145.79.227.50:4373 , piece_id=43V2KGMCF3E646EMPUNPREXNOZSGX3TYU4CRBNOXHJACCK457KEQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332690503 (597h56m38.203773548s) [noise=true , address=145.79.227.51:4410 , piece_id=2MUSHTPGNSDQRX4SZFJ7IHSWIP3YZJDP7EZLZPS43NQ4UBPBVGAQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.333563886 (597h56m38.204646921s) [noise=true , address=145.79.227.28:4726 , piece_id=FCDFQ72PEYOGOCG25C6UBKGYSM6IULNVEU6CWQ5VUA6QV5MXFODQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:39.956933212 (597h56m39.828016247s) [upload=524125]: Next returned nil segment Dec 30 13:25:39.956950835 (597h56m39.82803387s) [upload=524125]: Waiting for error group managing segments... Dec 30 13:25:40.325833913 (597h56m40.196916948s) [noise=true , address=145.79.227.50:4373 , piece_id=43V2KGMCF3E646EMPUNPREXNOZSGX3TYU4CRBNOXHJACCK457KEQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.345931001 (597h56m40.217014046s) [noise=true , address=145.79.227.12:4765 , piece_id=JI3TAHCJKJBVASYTI5IGZPF7EWWDTORGYJOER4NNNDGGS3JZHG6Q, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.347821541 (597h56m40.218904586s) [noise=true , address=145.79.227.45:4229 , piece_id=QOB22CQKW5NP2ZKY35JT7QFA4ZVQG7GRGDWXXJFCYSRQ55K7IKQQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.349394426 (597h56m40.220477461s) [noise=true , address=145.79.227.19:4079 , piece_id=TPPGIPDU2MHYFH4U57PN5PWRW767NKA2OLZVPMPLCDGG4GIKXOQQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.349754000 (597h56m40.220837035s) [noise=true , address=145.79.227.3:4149 , piece_id=7IQ5GONSR2A2AHH6TIWD2EIHAYFM3FV5VQJBHBZ7W54NFZYBHUXQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.350341288 (597h56m40.221424323s) [noise=true , address=145.79.227.28:4726 , piece_id=FCDFQ72PEYOGOCG25C6UBKGYSM6IULNVEU6CWQ5VUA6QV5MXFODQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.358245619 (597h56m40.229328665s) [noise=true , address=145.79.227.47:4265 , piece_id=6GQJA2CDH234RH3RA7Y4D643SUNLT5K44PFHNZ4UABMLPJVWKKKQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.363673435 (597h56m40.23475648s) [noise=true , address=145.79.227.13:4804 , piece_id=F23E2YSHHBJ3F6Y5VGNKXZYGSKK6KVV3TDBCDT5DGG5SI6J5YVVA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.408050705 (597h56m40.27913374s) [noise=true , address=145.79.227.51:4410 , piece_id=2MUSHTPGNSDQRX4SZFJ7IHSWIP3YZJDP7EZLZPS43NQ4UBPBVGAQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.475912424 (597h56m40.346995469s) [noise=true , address=145.79.227.52:4447 , piece_id=O7H7SBH24ADRRGIM6I5ZTAIO57SRL3MRBLDAOXJ6TUJIRKSVLEDA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.621656152 (597h56m40.492739187s) [noise=true , address=45.91.116.12:4226 , piece_id=NP2HBC7JM233MRXD4AEMGEHVPCJS45JH36DRI5F3JQSZFEINJZOA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.646459411 (597h56m40.517542456s) [noise=true , address=77.90.31.14:4664 , piece_id=TBG2LWSXBYVLOO456JJDS34O6CI5APU2J2JBONCGY2BQ72TWVUSQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.664606636 (597h56m40.535689671s) [noise=true , address=145.79.227.29:4943 , piece_id=MLF4KPYLI5GXMN76RJYFC24NU5J3GTDJZFEO3NYAMT5MVCMY4TOA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.664626454 (597h56m40.535709489s) [seg_pos={0 0}, upload=524125]: Segment reached optimal threshold of 13 pieces. Dec 30 13:25:40.664856752 (597h56m40.535939797s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.664866050 (597h56m40.535949085s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.664997029 (597h56m40.536080084s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.665008061 (597h56m40.536091096s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.665120865 (597h56m40.5362039s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.665128981 (597h56m40.536212016s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.665149460 (597h56m40.536232495s) [seg_pos={0 0}, upload=524125]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:25:40.665224904 (597h56m40.536307939s) [upload=524125]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:25:40.665231406 (597h56m40.536314451s) [upload=524125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:25:40.992575908 (597h56m40.863658943s) [upload=524125]: Done uploading segments... Dec 30 13:25:40.992593722 (597h56m40.863676757s) [upload=524125]: Upload finished. err: <nil> Duration: 8.177607645s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 16.896724647s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:26:34 UTC) orbiter warning: context deadline exceeded Duration: 16.896753071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [X.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXXXX.] duration: 3.740054275s trace: https://jaeger.dev.storj.io/trace/600bf0a72074e26e FAIL (2025-12-30 13:26:32 UTC) orbiter warning: context deadline exceeded Duration: 3.740079594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X..X.X..X....X........XX................X.XX....XX.] duration: 5.854249806s trace: https://jaeger.dev.storj.io/trace/3f971db36ddeca24 FAIL (2025-12-30 13:26:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 13:25:38.079780045 (597h56m37.95086308s) [upload=524125]: Starting upload Dec 30 13:25:38.079836973 (597h56m37.950920008s) [upload=524125]: Done starting upload Dec 30 13:25:38.080027135 (597h56m37.95111017s) [upload=524125]: Uploading segments... Dec 30 13:25:38.080066340 (597h56m37.951149376s) [upload=524125]: Got next segment. Inline: false Dec 30 13:25:38.080128628 (597h56m37.951211663s) [upload=524125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:25:38.080134641 (597h56m37.951217676s) [upload=524125]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:25:38.329259962 (597h56m38.200343017s) [seg_pos={0 0}, upload=524125]: Begin upload segment... Dec 30 13:25:38.329319876 (597h56m38.200402921s) [seg_pos={0 0}, upload=524125]: Done begin upload segment. Dec 30 13:25:38.329348531 (597h56m38.200431566s) [noise=true , address=145.79.227.47:4265 , piece_id=6GQJA2CDH234RH3RA7Y4D643SUNLT5K44PFHNZ4UABMLPJVWKKKQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.329365673 (597h56m38.200448708s) [noise=true , address=145.79.227.45:4229 , piece_id=QOB22CQKW5NP2ZKY35JT7QFA4ZVQG7GRGDWXXJFCYSRQ55K7IKQQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.329842341 (597h56m38.200925376s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330208908 (597h56m38.201291943s) [noise=true , address=45.91.116.12:4226 , piece_id=NP2HBC7JM233MRXD4AEMGEHVPCJS45JH36DRI5F3JQSZFEINJZOA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330396787 (597h56m38.201479822s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330724059 (597h56m38.201807095s) [noise=true , address=145.79.227.13:4804 , piece_id=F23E2YSHHBJ3F6Y5VGNKXZYGSKK6KVV3TDBCDT5DGG5SI6J5YVVA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331058716 (597h56m38.202141761s) [noise=true , address=77.90.31.14:4664 , piece_id=TBG2LWSXBYVLOO456JJDS34O6CI5APU2J2JBONCGY2BQ72TWVUSQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331433981 (597h56m38.202517016s) [noise=true , address=145.79.227.3:4149 , piece_id=7IQ5GONSR2A2AHH6TIWD2EIHAYFM3FV5VQJBHBZ7W54NFZYBHUXQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331567595 (597h56m38.202650631s) [noise=true , address=145.79.227.12:4765 , piece_id=JI3TAHCJKJBVASYTI5IGZPF7EWWDTORGYJOER4NNNDGGS3JZHG6Q, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331754090 (597h56m38.202837125s) [noise=true , address=145.79.227.19:4079 , piece_id=TPPGIPDU2MHYFH4U57PN5PWRW767NKA2OLZVPMPLCDGG4GIKXOQQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331958239 (597h56m38.203041274s) [noise=true , address=145.79.227.29:4943 , piece_id=MLF4KPYLI5GXMN76RJYFC24NU5J3GTDJZFEO3NYAMT5MVCMY4TOA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332059652 (597h56m38.203142687s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332380894 (597h56m38.203463929s) [noise=true , address=145.79.227.52:4447 , piece_id=O7H7SBH24ADRRGIM6I5ZTAIO57SRL3MRBLDAOXJ6TUJIRKSVLEDA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332540628 (597h56m38.203623663s) [noise=true , address=145.79.227.50:4373 , piece_id=43V2KGMCF3E646EMPUNPREXNOZSGX3TYU4CRBNOXHJACCK457KEQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332690503 (597h56m38.203773548s) [noise=true , address=145.79.227.51:4410 , piece_id=2MUSHTPGNSDQRX4SZFJ7IHSWIP3YZJDP7EZLZPS43NQ4UBPBVGAQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.333563886 (597h56m38.204646921s) [noise=true , address=145.79.227.28:4726 , piece_id=FCDFQ72PEYOGOCG25C6UBKGYSM6IULNVEU6CWQ5VUA6QV5MXFODQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:39.956933212 (597h56m39.828016247s) [upload=524125]: Next returned nil segment Dec 30 13:25:39.956950835 (597h56m39.82803387s) [upload=524125]: Waiting for error group managing segments... Dec 30 13:25:40.325833913 (597h56m40.196916948s) [noise=true , address=145.79.227.50:4373 , piece_id=43V2KGMCF3E646EMPUNPREXNOZSGX3TYU4CRBNOXHJACCK457KEQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.345931001 (597h56m40.217014046s) [noise=true , address=145.79.227.12:4765 , piece_id=JI3TAHCJKJBVASYTI5IGZPF7EWWDTORGYJOER4NNNDGGS3JZHG6Q, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.347821541 (597h56m40.218904586s) [noise=true , address=145.79.227.45:4229 , piece_id=QOB22CQKW5NP2ZKY35JT7QFA4ZVQG7GRGDWXXJFCYSRQ55K7IKQQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.349394426 (597h56m40.220477461s) [noise=true , address=145.79.227.19:4079 , piece_id=TPPGIPDU2MHYFH4U57PN5PWRW767NKA2OLZVPMPLCDGG4GIKXOQQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.349754000 (597h56m40.220837035s) [noise=true , address=145.79.227.3:4149 , piece_id=7IQ5GONSR2A2AHH6TIWD2EIHAYFM3FV5VQJBHBZ7W54NFZYBHUXQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.350341288 (597h56m40.221424323s) [noise=true , address=145.79.227.28:4726 , piece_id=FCDFQ72PEYOGOCG25C6UBKGYSM6IULNVEU6CWQ5VUA6QV5MXFODQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.358245619 (597h56m40.229328665s) [noise=true , address=145.79.227.47:4265 , piece_id=6GQJA2CDH234RH3RA7Y4D643SUNLT5K44PFHNZ4UABMLPJVWKKKQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.363673435 (597h56m40.23475648s) [noise=true , address=145.79.227.13:4804 , piece_id=F23E2YSHHBJ3F6Y5VGNKXZYGSKK6KVV3TDBCDT5DGG5SI6J5YVVA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.408050705 (597h56m40.27913374s) [noise=true , address=145.79.227.51:4410 , piece_id=2MUSHTPGNSDQRX4SZFJ7IHSWIP3YZJDP7EZLZPS43NQ4UBPBVGAQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.475912424 (597h56m40.346995469s) [noise=true , address=145.79.227.52:4447 , piece_id=O7H7SBH24ADRRGIM6I5ZTAIO57SRL3MRBLDAOXJ6TUJIRKSVLEDA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.621656152 (597h56m40.492739187s) [noise=true , address=45.91.116.12:4226 , piece_id=NP2HBC7JM233MRXD4AEMGEHVPCJS45JH36DRI5F3JQSZFEINJZOA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.646459411 (597h56m40.517542456s) [noise=true , address=77.90.31.14:4664 , piece_id=TBG2LWSXBYVLOO456JJDS34O6CI5APU2J2JBONCGY2BQ72TWVUSQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.664606636 (597h56m40.535689671s) [noise=true , address=145.79.227.29:4943 , piece_id=MLF4KPYLI5GXMN76RJYFC24NU5J3GTDJZFEO3NYAMT5MVCMY4TOA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.664626454 (597h56m40.535709489s) [seg_pos={0 0}, upload=524125]: Segment reached optimal threshold of 13 pieces. Dec 30 13:25:40.664856752 (597h56m40.535939797s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.664866050 (597h56m40.535949085s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.664997029 (597h56m40.536080084s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.665008061 (597h56m40.536091096s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.665120865 (597h56m40.5362039s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.665128981 (597h56m40.536212016s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.665149460 (597h56m40.536232495s) [seg_pos={0 0}, upload=524125]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:25:40.665224904 (597h56m40.536307939s) [upload=524125]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:25:40.665231406 (597h56m40.536314451s) [upload=524125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:25:40.992575908 (597h56m40.863658943s) [upload=524125]: Done uploading segments... Dec 30 13:25:40.992593722 (597h56m40.863676757s) [upload=524125]: Upload finished. err: <nil> Duration: 5.854567401s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X........X...........X..X........X...X...XXX.......X...XX.X] duration: 30.000960625s trace: https://jaeger.dev.storj.io/trace/4408905d00d478d9 FAIL (2025-12-30 13:26:12 UTC) orbiter warning: context deadline exceeded Duration: 30.000982477s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X..X.X..X....X........XX................X.XX....XX] duration: 30.003230215s trace: https://jaeger.dev.storj.io/trace/3f971db36ddeca24 FAIL (2025-12-30 13:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 13:25:38.079780045 (597h56m37.95086308s) [upload=524125]: Starting upload Dec 30 13:25:38.079836973 (597h56m37.950920008s) [upload=524125]: Done starting upload Dec 30 13:25:38.080027135 (597h56m37.95111017s) [upload=524125]: Uploading segments... Dec 30 13:25:38.080066340 (597h56m37.951149376s) [upload=524125]: Got next segment. Inline: false Dec 30 13:25:38.080128628 (597h56m37.951211663s) [upload=524125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:25:38.080134641 (597h56m37.951217676s) [upload=524125]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:25:38.329259962 (597h56m38.200343017s) [seg_pos={0 0}, upload=524125]: Begin upload segment... Dec 30 13:25:38.329319876 (597h56m38.200402921s) [seg_pos={0 0}, upload=524125]: Done begin upload segment. Dec 30 13:25:38.329348531 (597h56m38.200431566s) [noise=true , address=145.79.227.47:4265 , piece_id=6GQJA2CDH234RH3RA7Y4D643SUNLT5K44PFHNZ4UABMLPJVWKKKQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.329365673 (597h56m38.200448708s) [noise=true , address=145.79.227.45:4229 , piece_id=QOB22CQKW5NP2ZKY35JT7QFA4ZVQG7GRGDWXXJFCYSRQ55K7IKQQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.329842341 (597h56m38.200925376s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330208908 (597h56m38.201291943s) [noise=true , address=45.91.116.12:4226 , piece_id=NP2HBC7JM233MRXD4AEMGEHVPCJS45JH36DRI5F3JQSZFEINJZOA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330396787 (597h56m38.201479822s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.330724059 (597h56m38.201807095s) [noise=true , address=145.79.227.13:4804 , piece_id=F23E2YSHHBJ3F6Y5VGNKXZYGSKK6KVV3TDBCDT5DGG5SI6J5YVVA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331058716 (597h56m38.202141761s) [noise=true , address=77.90.31.14:4664 , piece_id=TBG2LWSXBYVLOO456JJDS34O6CI5APU2J2JBONCGY2BQ72TWVUSQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331433981 (597h56m38.202517016s) [noise=true , address=145.79.227.3:4149 , piece_id=7IQ5GONSR2A2AHH6TIWD2EIHAYFM3FV5VQJBHBZ7W54NFZYBHUXQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331567595 (597h56m38.202650631s) [noise=true , address=145.79.227.12:4765 , piece_id=JI3TAHCJKJBVASYTI5IGZPF7EWWDTORGYJOER4NNNDGGS3JZHG6Q, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331754090 (597h56m38.202837125s) [noise=true , address=145.79.227.19:4079 , piece_id=TPPGIPDU2MHYFH4U57PN5PWRW767NKA2OLZVPMPLCDGG4GIKXOQQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.331958239 (597h56m38.203041274s) [noise=true , address=145.79.227.29:4943 , piece_id=MLF4KPYLI5GXMN76RJYFC24NU5J3GTDJZFEO3NYAMT5MVCMY4TOA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332059652 (597h56m38.203142687s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332380894 (597h56m38.203463929s) [noise=true , address=145.79.227.52:4447 , piece_id=O7H7SBH24ADRRGIM6I5ZTAIO57SRL3MRBLDAOXJ6TUJIRKSVLEDA, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332540628 (597h56m38.203623663s) [noise=true , address=145.79.227.50:4373 , piece_id=43V2KGMCF3E646EMPUNPREXNOZSGX3TYU4CRBNOXHJACCK457KEQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.332690503 (597h56m38.203773548s) [noise=true , address=145.79.227.51:4410 , piece_id=2MUSHTPGNSDQRX4SZFJ7IHSWIP3YZJDP7EZLZPS43NQ4UBPBVGAQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:38.333563886 (597h56m38.204646921s) [noise=true , address=145.79.227.28:4726 , piece_id=FCDFQ72PEYOGOCG25C6UBKGYSM6IULNVEU6CWQ5VUA6QV5MXFODQ, seg_pos={0 0}, upload=524125]: Uploading piece... Dec 30 13:25:39.956933212 (597h56m39.828016247s) [upload=524125]: Next returned nil segment Dec 30 13:25:39.956950835 (597h56m39.82803387s) [upload=524125]: Waiting for error group managing segments... Dec 30 13:25:40.325833913 (597h56m40.196916948s) [noise=true , address=145.79.227.50:4373 , piece_id=43V2KGMCF3E646EMPUNPREXNOZSGX3TYU4CRBNOXHJACCK457KEQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.345931001 (597h56m40.217014046s) [noise=true , address=145.79.227.12:4765 , piece_id=JI3TAHCJKJBVASYTI5IGZPF7EWWDTORGYJOER4NNNDGGS3JZHG6Q, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.347821541 (597h56m40.218904586s) [noise=true , address=145.79.227.45:4229 , piece_id=QOB22CQKW5NP2ZKY35JT7QFA4ZVQG7GRGDWXXJFCYSRQ55K7IKQQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.349394426 (597h56m40.220477461s) [noise=true , address=145.79.227.19:4079 , piece_id=TPPGIPDU2MHYFH4U57PN5PWRW767NKA2OLZVPMPLCDGG4GIKXOQQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.349754000 (597h56m40.220837035s) [noise=true , address=145.79.227.3:4149 , piece_id=7IQ5GONSR2A2AHH6TIWD2EIHAYFM3FV5VQJBHBZ7W54NFZYBHUXQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.350341288 (597h56m40.221424323s) [noise=true , address=145.79.227.28:4726 , piece_id=FCDFQ72PEYOGOCG25C6UBKGYSM6IULNVEU6CWQ5VUA6QV5MXFODQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.358245619 (597h56m40.229328665s) [noise=true , address=145.79.227.47:4265 , piece_id=6GQJA2CDH234RH3RA7Y4D643SUNLT5K44PFHNZ4UABMLPJVWKKKQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.363673435 (597h56m40.23475648s) [noise=true , address=145.79.227.13:4804 , piece_id=F23E2YSHHBJ3F6Y5VGNKXZYGSKK6KVV3TDBCDT5DGG5SI6J5YVVA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.408050705 (597h56m40.27913374s) [noise=true , address=145.79.227.51:4410 , piece_id=2MUSHTPGNSDQRX4SZFJ7IHSWIP3YZJDP7EZLZPS43NQ4UBPBVGAQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.475912424 (597h56m40.346995469s) [noise=true , address=145.79.227.52:4447 , piece_id=O7H7SBH24ADRRGIM6I5ZTAIO57SRL3MRBLDAOXJ6TUJIRKSVLEDA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.621656152 (597h56m40.492739187s) [noise=true , address=45.91.116.12:4226 , piece_id=NP2HBC7JM233MRXD4AEMGEHVPCJS45JH36DRI5F3JQSZFEINJZOA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.646459411 (597h56m40.517542456s) [noise=true , address=77.90.31.14:4664 , piece_id=TBG2LWSXBYVLOO456JJDS34O6CI5APU2J2JBONCGY2BQ72TWVUSQ, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.664606636 (597h56m40.535689671s) [noise=true , address=145.79.227.29:4943 , piece_id=MLF4KPYLI5GXMN76RJYFC24NU5J3GTDJZFEO3NYAMT5MVCMY4TOA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: <nil> Dec 30 13:25:40.664626454 (597h56m40.535709489s) [seg_pos={0 0}, upload=524125]: Segment reached optimal threshold of 13 pieces. Dec 30 13:25:40.664856752 (597h56m40.535939797s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.664866050 (597h56m40.535949085s) [noise=true , address=145.79.227.49:4337 , piece_id=VZXE2XVVTU6ZKHRP5YYAE7N3HJFALBILCQW2EOXAGG72KNGSS6LA, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.664997029 (597h56m40.536080084s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.665008061 (597h56m40.536091096s) [noise=true , address=45.91.116.8:4046 , piece_id=2NUJU7DG27B2YUDUBWJ3GS2NSDPOYAX5J4VYKA7W5F4J2B7XY37Q, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.665120865 (597h56m40.5362039s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:40.665128981 (597h56m40.536212016s) [noise=true , address=145.79.227.35:4481 , piece_id=XKMIGCDTKNSEL5I5RRG2LHLHDYGKQ654KFTKXS2DPK4JNOKCRINA, seg_pos={0 0}, upload=524125]: Long tail cancellation due to optimal threshold Dec 30 13:25:40.665149460 (597h56m40.536232495s) [seg_pos={0 0}, upload=524125]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:25:40.665224904 (597h56m40.536307939s) [upload=524125]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:25:40.665231406 (597h56m40.536314451s) [upload=524125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:25:40.992575908 (597h56m40.863658943s) [upload=524125]: Done uploading segments... Dec 30 13:25:40.992593722 (597h56m40.863676757s) [upload=524125]: Upload finished. err: <nil> Duration: 30.00324325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X....X........X..............X.......X...X..............] duration: 43.636943636s trace: https://jaeger.dev.storj.io/trace/327137fdf99c8cd2 FAIL (2025-12-30 13:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463267094 (597h44m10.334350129s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463636957 (597h44m10.334719992s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463733021 (597h44m10.334816076s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463967257 (597h44m10.335050292s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464317554 (597h44m10.335400589s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464332772 (597h44m10.335415817s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464747913 (597h44m10.335830958s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464994803 (597h44m10.336077848s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.465113149 (597h44m10.336196184s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.465730814 (597h44m10.336813859s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.466479270 (597h44m10.337562315s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.466845497 (597h44m10.337928542s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.467111013 (597h44m10.338194058s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.467663886 (597h44m10.338746921s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:11.913537776 (597h44m11.784620821s) [upload=523949]: Next returned nil segment Dec 30 13:13:11.913553607 (597h44m11.784636652s) [upload=523949]: Waiting for error group managing segments... Dec 30 13:13:12.310633846 (597h44m12.181716891s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.369636326 (597h44m12.240719361s) [noise=true , address=145.79.227.5:4248 , piece_id=GVI3MNAX2S2H6IKO67GTHYD6Z2FXPDPNRRSFEBRICJKOZYOP6BLA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.489444639 (597h44m12.360527674s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.552298983 (597h44m12.423382018s) [noise=true , address=64.49.13.27:4967 , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.637395539 (597h44m12.508478574s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.647923005 (597h44m12.51900604s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.718170647 (597h44m12.589253692s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.731374770 (597h44m12.602457815s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.792347862 (597h44m12.663430917s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.202419249 (597h44m13.073502284s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.235840199 (597h44m13.106923234s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.322780525 (597h44m13.193863571s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.442893078 (597h44m13.313976113s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.631615082 (597h44m13.502698127s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.631640270 (597h44m13.502723315s) [seg_pos={0 0}, upload=523949]: Segment reached optimal threshold of 14 pieces. Dec 30 13:13:13.631902971 (597h44m13.502986036s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF): protocol: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection Dec 30 13:13:13.631912879 (597h44m13.502995935s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632041715 (597h44m13.50312475s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 30 13:13:13.632057996 (597h44m13.503141031s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632602493 (597h44m13.503685528s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection Dec 30 13:13:13.632610257 (597h44m13.503693302s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632858089 (597h44m13.503941134s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 30 13:13:13.632864872 (597h44m13.503947907s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632959923 (597h44m13.504042958s) [seg_pos={0 0}, upload=523949]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:13:13.633117143 (597h44m13.504200188s) [upload=523949]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:13:13.633123475 (597h44m13.50420651s) [upload=523949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:13:13.920616664 (597h44m13.791699709s) [upload=523949]: Done uploading segments... Dec 30 13:13:13.920640600 (597h44m13.791723645s) [upload=523949]: Upload finished. err: <nil> Duration: 43.636958124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXXXX] duration: 5.000962687s trace: https://jaeger.dev.storj.io/trace/600bf0a72074e26e FAIL (2025-12-30 13:26:06 UTC) orbiter warning: context deadline exceeded Duration: 5.000989628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 14.265214098s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:25:59 UTC) orbiter warning: context deadline exceeded Duration: 14.265242752s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X........X...........X..X........X...X...XXX.......X...XX.] duration: 30.000441372s trace: https://jaeger.dev.storj.io/trace/4513c59973ee1652 FAIL (2025-12-30 13:25:38 UTC) orbiter warning: context deadline exceeded Duration: 30.000503349s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...........X..X.X..X....X........XX................X.XX....X] duration: 30.002362533s trace: https://jaeger.dev.storj.io/trace/7620a8a821a6beb9 FAIL (2025-12-30 13:25:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: MVSB24VLELKLU4VHB7MWOIFGDDGSQJB2NIHAFBFXLEEW7UQIJLBQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: BNB6SEDJKXJTWCEZXU2N7CTQXGKJ5TDR6RZOC3CEXNZTXTF4GCXQ) context deadline exceeded ======LOGS====== Dec 30 13:25:07.847229621 (597h56m7.718312666s) [upload=524117]: Starting upload Dec 30 13:25:07.847292702 (597h56m7.718375737s) [upload=524117]: Done starting upload Dec 30 13:25:07.847479046 (597h56m7.718562091s) [upload=524117]: Uploading segments... Dec 30 13:25:07.847523592 (597h56m7.718606627s) [upload=524117]: Got next segment. Inline: false Dec 30 13:25:07.847557316 (597h56m7.718640361s) [upload=524117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:25:07.847562015 (597h56m7.71864505s) [upload=524117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:25:08.094611614 (597h56m7.965694669s) [seg_pos={0 0}, upload=524117]: Begin upload segment... Dec 30 13:25:08.094676528 (597h56m7.965759573s) [seg_pos={0 0}, upload=524117]: Done begin upload segment. Dec 30 13:25:08.094710212 (597h56m7.965793257s) [noise=true , address=145.79.227.42:4123 , piece_id=ARTHSZJJBWJKUET4P6XCCUNMQ4R272ZSQFYPFSKLUBTE45Y5ZZ6A, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.094913850 (597h56m7.965996885s) [noise=true , address=145.79.227.17:4971 , piece_id=BO72I5FHWAKM23PKDMBZ7PXE2YZF5EWKSO6Q7OKB7DJUACA4J3XQ, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.095546895 (597h56m7.96662993s) [noise=true , address=145.79.227.51:4411 , piece_id=S77MJOKOP4VPQTG5DGRXEWSLP6RGNHBBOGOE7NKH7Y2ZL6RRONCA, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.095978366 (597h56m7.967061411s) [noise=true , address=45.91.116.13:4155 , piece_id=V4GVTBEPOEON2NC6UZ3WAX5ZBZA4MWRNTSI2YX2Z7CUJHTLBVP7Q, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.096384249 (597h56m7.967467284s) [noise=true , address=145.79.227.48:4301 , piece_id=AY5WTZ2R52E2L7L7QRPXA7QN53OOH5FYXRJBJ3PNJOAYDSVEAMTQ, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.096728365 (597h56m7.9678114s) [noise=true , address=145.79.227.38:4977 , piece_id=KRH4CRV5J7AYXUZ5DZPXOTNMJSFXAA3ZCODNX455BASCHFIUQS5A, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.097066539 (597h56m7.968149584s) [noise=true , address=145.79.227.6:4259 , piece_id=J7ROGZBVPMG5ZYQ7V2HCLBWAA6EU64YJ5VVAUM6EC7SWZFIJXHLQ, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.097627607 (597h56m7.968710642s) [noise=true , address=145.79.227.52:4447 , piece_id=UEETZLEV5OFKWTSBJVJ6ENDS62ECO2XFNTZ2JJNCAD5BTNVVQJHA, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.098198153 (597h56m7.969281228s) [noise=true , address=77.90.31.10:4518 , piece_id=MVSB24VLELKLU4VHB7MWOIFGDDGSQJB2NIHAFBFXLEEW7UQIJLBQ, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.099464033 (597h56m7.970547078s) [noise=true , address=145.79.227.39:4015 , piece_id=BHBQBQWMYQRPGRUWTVUGH24E4UQZ56YNOLXC4RXKXMULQ2AI4QKA, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.101090611 (597h56m7.972173656s) [noise=true , address=45.91.116.6:4974 , piece_id=BNB6SEDJKXJTWCEZXU2N7CTQXGKJ5TDR6RZOC3CEXNZTXTF4GCXQ, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.102974969 (597h56m7.974058014s) [noise=true , address=77.90.31.13:4629 , piece_id=XJT452NPWNONNJGUE7F4PGEDF4BX3LPI6YJGXSLRF65D7ZV7ZTXQ, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.104842175 (597h56m7.97592521s) [noise=true , address=145.79.227.11:4729 , piece_id=GVRXKQ3A3RCYHUZAI3QRF4MGY3OPSWYVK3V7DU4W2XL4GUWMD34A, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.106361187 (597h56m7.977444232s) [noise=true , address=145.79.227.25:4617 , piece_id=LFZTXVIHROQEUOFQDRG6BMXHYMX7HMI3CQYIMQVUNLJ6GTF3LQ2A, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.107805467 (597h56m7.978888512s) [noise=true , address=145.79.227.5:4223 , piece_id=GTPINNANZ7GQSQYKZMQXWWXG2YG5TLOETKJHYEILSCPZ7CZWTTLQ, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:08.109213278 (597h56m7.980296323s) [noise=true , address=45.91.116.12:4225 , piece_id=XV5TDUTWKRSB6XWDPDQH5PXAFT7GD3ASHN4XZCRZ527FKGZYZ6ZQ, seg_pos={0 0}, upload=524117]: Uploading piece... Dec 30 13:25:09.736320872 (597h56m9.607403917s) [upload=524117]: Next returned nil segment Dec 30 13:25:09.736338666 (597h56m9.607421711s) [upload=524117]: Waiting for error group managing segments... Dec 30 13:25:10.101063306 (597h56m9.972146341s) [noise=true , address=145.79.227.5:4223 , piece_id=GTPINNANZ7GQSQYKZMQXWWXG2YG5TLOETKJHYEILSCPZ7CZWTTLQ, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.110511887 (597h56m9.981594922s) [noise=true , address=145.79.227.38:4977 , piece_id=KRH4CRV5J7AYXUZ5DZPXOTNMJSFXAA3ZCODNX455BASCHFIUQS5A, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.111141646 (597h56m9.982224701s) [noise=true , address=145.79.227.51:4411 , piece_id=S77MJOKOP4VPQTG5DGRXEWSLP6RGNHBBOGOE7NKH7Y2ZL6RRONCA, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.111598355 (597h56m9.982681391s) [noise=true , address=145.79.227.17:4971 , piece_id=BO72I5FHWAKM23PKDMBZ7PXE2YZF5EWKSO6Q7OKB7DJUACA4J3XQ, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.114112963 (597h56m9.985195999s) [noise=true , address=145.79.227.6:4259 , piece_id=J7ROGZBVPMG5ZYQ7V2HCLBWAA6EU64YJ5VVAUM6EC7SWZFIJXHLQ, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.122427265 (597h56m9.993510311s) [noise=true , address=145.79.227.11:4729 , piece_id=GVRXKQ3A3RCYHUZAI3QRF4MGY3OPSWYVK3V7DU4W2XL4GUWMD34A, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.142929896 (597h56m10.014012931s) [noise=true , address=145.79.227.25:4617 , piece_id=LFZTXVIHROQEUOFQDRG6BMXHYMX7HMI3CQYIMQVUNLJ6GTF3LQ2A, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.162236188 (597h56m10.033319233s) [noise=true , address=145.79.227.42:4123 , piece_id=ARTHSZJJBWJKUET4P6XCCUNMQ4R272ZSQFYPFSKLUBTE45Y5ZZ6A, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.204652335 (597h56m10.07573537s) [noise=true , address=145.79.227.39:4015 , piece_id=BHBQBQWMYQRPGRUWTVUGH24E4UQZ56YNOLXC4RXKXMULQ2AI4QKA, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.205784551 (597h56m10.076867586s) [noise=true , address=145.79.227.48:4301 , piece_id=AY5WTZ2R52E2L7L7QRPXA7QN53OOH5FYXRJBJ3PNJOAYDSVEAMTQ, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.214709476 (597h56m10.085792521s) [noise=true , address=145.79.227.52:4447 , piece_id=UEETZLEV5OFKWTSBJVJ6ENDS62ECO2XFNTZ2JJNCAD5BTNVVQJHA, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.281442004 (597h56m10.152525049s) [noise=true , address=77.90.31.10:4518 , piece_id=MVSB24VLELKLU4VHB7MWOIFGDDGSQJB2NIHAFBFXLEEW7UQIJLBQ, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.498116874 (597h56m10.369199919s) [noise=true , address=45.91.116.6:4974 , piece_id=BNB6SEDJKXJTWCEZXU2N7CTQXGKJ5TDR6RZOC3CEXNZTXTF4GCXQ, seg_pos={0 0}, upload=524117]: Done uploading piece. err: <nil> Dec 30 13:25:10.498138606 (597h56m10.369221641s) [seg_pos={0 0}, upload=524117]: Segment reached optimal threshold of 13 pieces. Dec 30 13:25:10.498357462 (597h56m10.369440497s) [noise=true , address=45.91.116.13:4155 , piece_id=V4GVTBEPOEON2NC6UZ3WAX5ZBZA4MWRNTSI2YX2Z7CUJHTLBVP7Q, seg_pos={0 0}, upload=524117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:10.498365557 (597h56m10.369448592s) [noise=true , address=45.91.116.13:4155 , piece_id=V4GVTBEPOEON2NC6UZ3WAX5ZBZA4MWRNTSI2YX2Z7CUJHTLBVP7Q, seg_pos={0 0}, upload=524117]: Long tail cancellation due to optimal threshold Dec 30 13:25:10.498371129 (597h56m10.369454164s) [noise=true , address=77.90.31.13:4629 , piece_id=XJT452NPWNONNJGUE7F4PGEDF4BX3LPI6YJGXSLRF65D7ZV7ZTXQ, seg_pos={0 0}, upload=524117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:10.498380436 (597h56m10.369463481s) [noise=true , address=77.90.31.13:4629 , piece_id=XJT452NPWNONNJGUE7F4PGEDF4BX3LPI6YJGXSLRF65D7ZV7ZTXQ, seg_pos={0 0}, upload=524117]: Long tail cancellation due to optimal threshold Dec 30 13:25:10.498521375 (597h56m10.36960441s) [noise=true , address=45.91.116.12:4225 , piece_id=XV5TDUTWKRSB6XWDPDQH5PXAFT7GD3ASHN4XZCRZ527FKGZYZ6ZQ, seg_pos={0 0}, upload=524117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 13:25:10.498529851 (597h56m10.369612896s) [noise=true , address=45.91.116.12:4225 , piece_id=XV5TDUTWKRSB6XWDPDQH5PXAFT7GD3ASHN4XZCRZ527FKGZYZ6ZQ, seg_pos={0 0}, upload=524117]: Long tail cancellation due to optimal threshold Dec 30 13:25:10.498545862 (597h56m10.369628897s) [seg_pos={0 0}, upload=524117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:25:10.498618660 (597h56m10.369701695s) [upload=524117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:25:10.498624100 (597h56m10.369707145s) [upload=524117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:25:10.788323190 (597h56m10.659406225s) [upload=524117]: Done uploading segments... Dec 30 13:25:10.788345092 (597h56m10.659428127s) [upload=524117]: Upload finished. err: <nil> Duration: 30.002380958s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration: 32.127283999s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:25:26 UTC) orbiter warning: context deadline exceeded Duration: 32.127327112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXXX] duration: 5.000807845s trace: https://jaeger.dev.storj.io/trace/12e362a5eb8dd968 FAIL (2025-12-30 13:25:24 UTC) orbiter warning: context deadline exceeded Duration: 5.00082608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X....X........X..............X.......X...X.............] duration: 44.841515343s trace: https://jaeger.dev.storj.io/trace/327137fdf99c8cd2 FAIL (2025-12-30 13:25:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463267094 (597h44m10.334350129s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463636957 (597h44m10.334719992s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463733021 (597h44m10.334816076s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463967257 (597h44m10.335050292s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464317554 (597h44m10.335400589s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464332772 (597h44m10.335415817s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464747913 (597h44m10.335830958s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464994803 (597h44m10.336077848s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.465113149 (597h44m10.336196184s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.465730814 (597h44m10.336813859s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.466479270 (597h44m10.337562315s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.466845497 (597h44m10.337928542s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.467111013 (597h44m10.338194058s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.467663886 (597h44m10.338746921s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:11.913537776 (597h44m11.784620821s) [upload=523949]: Next returned nil segment Dec 30 13:13:11.913553607 (597h44m11.784636652s) [upload=523949]: Waiting for error group managing segments... Dec 30 13:13:12.310633846 (597h44m12.181716891s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.369636326 (597h44m12.240719361s) [noise=true , address=145.79.227.5:4248 , piece_id=GVI3MNAX2S2H6IKO67GTHYD6Z2FXPDPNRRSFEBRICJKOZYOP6BLA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.489444639 (597h44m12.360527674s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.552298983 (597h44m12.423382018s) [noise=true , address=64.49.13.27:4967 , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.637395539 (597h44m12.508478574s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.647923005 (597h44m12.51900604s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.718170647 (597h44m12.589253692s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.731374770 (597h44m12.602457815s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.792347862 (597h44m12.663430917s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.202419249 (597h44m13.073502284s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.235840199 (597h44m13.106923234s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.322780525 (597h44m13.193863571s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.442893078 (597h44m13.313976113s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.631615082 (597h44m13.502698127s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.631640270 (597h44m13.502723315s) [seg_pos={0 0}, upload=523949]: Segment reached optimal threshold of 14 pieces. Dec 30 13:13:13.631902971 (597h44m13.502986036s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF): protocol: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection Dec 30 13:13:13.631912879 (597h44m13.502995935s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632041715 (597h44m13.50312475s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 30 13:13:13.632057996 (597h44m13.503141031s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632602493 (597h44m13.503685528s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection Dec 30 13:13:13.632610257 (597h44m13.503693302s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632858089 (597h44m13.503941134s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 30 13:13:13.632864872 (597h44m13.503947907s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632959923 (597h44m13.504042958s) [seg_pos={0 0}, upload=523949]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:13:13.633117143 (597h44m13.504200188s) [upload=523949]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:13:13.633123475 (597h44m13.50420651s) [upload=523949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:13:13.920616664 (597h44m13.791699709s) [upload=523949]: Done uploading segments... Dec 30 13:13:13.920640600 (597h44m13.791723645s) [upload=523949]: Upload finished. err: <nil> Duration: 44.841548746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X........X...........X..X........X...X...XXX.......X...XX] duration: 30.001090083s trace: https://jaeger.dev.storj.io/trace/4513c59973ee1652 FAIL (2025-12-30 13:24:53 UTC) orbiter warning: context deadline exceeded Duration: 30.001112326s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............X..X.X..X....X........XX................X.XX....] duration: 6.153234078s trace: https://jaeger.dev.storj.io/trace/1a1439816529e26e FAIL (2025-12-30 13:24:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA) context deadline exceeded ======LOGS====== Dec 30 13:21:29.510349015 (597h52m29.38143205s) [upload=524063]: Starting upload Dec 30 13:21:29.510459265 (597h52m29.3815423s) [upload=524063]: Done starting upload Dec 30 13:21:29.510558494 (597h52m29.381641529s) [upload=524063]: Uploading segments... Dec 30 13:21:29.510712408 (597h52m29.381795453s) [upload=524063]: Got next segment. Inline: false Dec 30 13:21:29.510754197 (597h52m29.381837242s) [upload=524063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:21:29.510759327 (597h52m29.381842362s) [upload=524063]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:21:29.762893408 (597h52m29.633976443s) [seg_pos={0 0}, upload=524063]: Begin upload segment... Dec 30 13:21:29.762954584 (597h52m29.634037629s) [seg_pos={0 0}, upload=524063]: Done begin upload segment. Dec 30 13:21:29.763018997 (597h52m29.634102032s) [noise=true , address=145.79.227.11:4731 , piece_id=BFM7VAM5JIMZXXTGSOJGU4CR2BTS22DCDAS4U3DZ3VOJGUJJLOHA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.763062069 (597h52m29.634145114s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.763987811 (597h52m29.635070856s) [noise=true , address=77.90.31.14:4661 , piece_id=YW4B33YWRWPXCFWRKOS57ATK5MXPRC2MLXFCCRBEAE4UH3S3H6FQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.764880712 (597h52m29.635963747s) [noise=true , address=145.79.227.42:4122 , piece_id=SZM3BNYSV3FRGT2E2WACH6DJCCT66NIHJELUT4WKWPOZKM2YG4BQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.765478029 (597h52m29.636561074s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.765723596 (597h52m29.636806632s) [noise=true , address=145.79.227.16:4911 , piece_id=YMISM7TAOTUSZ6F3NUPFB5EHC4SCRZFBEKKEFPOG5SOWNJSFGVJA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.766612329 (597h52m29.637695364s) [noise=true , address=145.79.227.39:4015 , piece_id=3ATCT6C4I66WDJNPLZ2ADP7M5YBNFS6UMWJWIJAE4HSGCHSDDHKQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.767873170 (597h52m29.638956205s) [noise=true , address=145.79.227.2:4115 , piece_id=5LEYO73EBV4FUBEPCCOIOHQJBVXR4AQCZDZ3SFGBQLU7LVNTVHGQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.768854288 (597h52m29.639937333s) [noise=true , address=145.79.227.24:4583 , piece_id=CP4VKRHNS6C2GLXATFVG6QKTUMFZIIZQN56HIQZKXLC7Y7X4VK5A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.769467075 (597h52m29.64055011s) [noise=true , address=145.79.227.50:4374 , piece_id=J7SYIFAINVSDERN3UWZIIZFKEBSVWKIETCYFJS3W2BFI6VVQIMTA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.770806114 (597h52m29.64188915s) [noise=true , address=145.79.227.26:4653 , piece_id=ODX2XWJTGCNCA6TCCAJEWRCX2BJF2XKYT2EYE5UWGN2SR7OJFA5A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.771265229 (597h52m29.642348265s) [noise=true , address=145.79.227.19:4078 , piece_id=JRNJ6NQPPOQL42JE6P2ABL6JF5EZCZ74R4DYCKZ5DXVGY5BDLKGQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.772594400 (597h52m29.643677445s) [noise=true , address=77.90.31.11:4553 , piece_id=DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.780612017 (597h52m29.651695052s) [noise=true , address=45.91.116.13:4153 , piece_id=7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.783374417 (597h52m29.654457462s) [noise=true , address=145.79.227.41:4085 , piece_id=TPDBBLHTA43O7L4E325PSB2FC5ZYU3JETDE6HUYMYHY73NXOX4YA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.785132534 (597h52m29.656215579s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:31.472089583 (597h52m31.343172618s) [upload=524063]: Next returned nil segment Dec 30 13:21:31.472121153 (597h52m31.343204188s) [upload=524063]: Waiting for error group managing segments... Dec 30 13:21:31.752334330 (597h52m31.623417376s) [noise=true , address=145.79.227.19:4078 , piece_id=JRNJ6NQPPOQL42JE6P2ABL6JF5EZCZ74R4DYCKZ5DXVGY5BDLKGQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757288283 (597h52m31.628371328s) [noise=true , address=145.79.227.39:4015 , piece_id=3ATCT6C4I66WDJNPLZ2ADP7M5YBNFS6UMWJWIJAE4HSGCHSDDHKQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757559550 (597h52m31.628642595s) [noise=true , address=145.79.227.16:4911 , piece_id=YMISM7TAOTUSZ6F3NUPFB5EHC4SCRZFBEKKEFPOG5SOWNJSFGVJA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757889207 (597h52m31.628972242s) [noise=true , address=145.79.227.2:4115 , piece_id=5LEYO73EBV4FUBEPCCOIOHQJBVXR4AQCZDZ3SFGBQLU7LVNTVHGQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.761055006 (597h52m31.632138041s) [noise=true , address=145.79.227.11:4731 , piece_id=BFM7VAM5JIMZXXTGSOJGU4CR2BTS22DCDAS4U3DZ3VOJGUJJLOHA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.767159039 (597h52m31.638242074s) [noise=true , address=145.79.227.41:4085 , piece_id=TPDBBLHTA43O7L4E325PSB2FC5ZYU3JETDE6HUYMYHY73NXOX4YA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.771751394 (597h52m31.642834429s) [noise=true , address=145.79.227.42:4122 , piece_id=SZM3BNYSV3FRGT2E2WACH6DJCCT66NIHJELUT4WKWPOZKM2YG4BQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.789060603 (597h52m31.660143648s) [noise=true , address=145.79.227.24:4583 , piece_id=CP4VKRHNS6C2GLXATFVG6QKTUMFZIIZQN56HIQZKXLC7Y7X4VK5A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.790015191 (597h52m31.661098226s) [noise=true , address=145.79.227.50:4374 , piece_id=J7SYIFAINVSDERN3UWZIIZFKEBSVWKIETCYFJS3W2BFI6VVQIMTA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.853644600 (597h52m31.724727665s) [noise=true , address=145.79.227.26:4653 , piece_id=ODX2XWJTGCNCA6TCCAJEWRCX2BJF2XKYT2EYE5UWGN2SR7OJFA5A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.119951609 (597h52m31.991034654s) [noise=true , address=77.90.31.11:4553 , piece_id=DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.332028685 (597h52m32.20311173s) [noise=true , address=45.91.116.13:4153 , piece_id=7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.333908855 (597h52m32.20499189s) [noise=true , address=77.90.31.14:4661 , piece_id=YW4B33YWRWPXCFWRKOS57ATK5MXPRC2MLXFCCRBEAE4UH3S3H6FQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.333922632 (597h52m32.205005667s) [seg_pos={0 0}, upload=524063]: Segment reached optimal threshold of 13 pieces. Dec 30 13:21:32.334117362 (597h52m32.205200407s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334126250 (597h52m32.205209285s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334420179 (597h52m32.205503234s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334427603 (597h52m32.205510638s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334656860 (597h52m32.205739905s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334665336 (597h52m32.205748381s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334698228 (597h52m32.205781263s) [seg_pos={0 0}, upload=524063]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:21:32.334774003 (597h52m32.205857038s) [upload=524063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:21:32.334779222 (597h52m32.205862267s) [upload=524063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:21:32.625237599 (597h52m32.496320644s) [upload=524063]: Done uploading segments... Dec 30 13:21:32.625290089 (597h52m32.496373134s) [upload=524063]: Upload finished. err: <nil> Duration: 6.153296425s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXXX] duration: 5.00266286s trace: https://jaeger.dev.storj.io/trace/685459292c146300 FAIL (2025-12-30 13:24:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002686214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X....X..........................................] duration: 22.487653435s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:24:25 UTC) orbiter warning: context deadline exceeded Duration: 22.487680807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............X..X.X..X....X........XX................X.XX...] duration: 5.964359823s trace: https://jaeger.dev.storj.io/trace/1a1439816529e26e FAIL (2025-12-30 13:24:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA) context deadline exceeded ======LOGS====== Dec 30 13:21:29.510349015 (597h52m29.38143205s) [upload=524063]: Starting upload Dec 30 13:21:29.510459265 (597h52m29.3815423s) [upload=524063]: Done starting upload Dec 30 13:21:29.510558494 (597h52m29.381641529s) [upload=524063]: Uploading segments... Dec 30 13:21:29.510712408 (597h52m29.381795453s) [upload=524063]: Got next segment. Inline: false Dec 30 13:21:29.510754197 (597h52m29.381837242s) [upload=524063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:21:29.510759327 (597h52m29.381842362s) [upload=524063]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:21:29.762893408 (597h52m29.633976443s) [seg_pos={0 0}, upload=524063]: Begin upload segment... Dec 30 13:21:29.762954584 (597h52m29.634037629s) [seg_pos={0 0}, upload=524063]: Done begin upload segment. Dec 30 13:21:29.763018997 (597h52m29.634102032s) [noise=true , address=145.79.227.11:4731 , piece_id=BFM7VAM5JIMZXXTGSOJGU4CR2BTS22DCDAS4U3DZ3VOJGUJJLOHA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.763062069 (597h52m29.634145114s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.763987811 (597h52m29.635070856s) [noise=true , address=77.90.31.14:4661 , piece_id=YW4B33YWRWPXCFWRKOS57ATK5MXPRC2MLXFCCRBEAE4UH3S3H6FQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.764880712 (597h52m29.635963747s) [noise=true , address=145.79.227.42:4122 , piece_id=SZM3BNYSV3FRGT2E2WACH6DJCCT66NIHJELUT4WKWPOZKM2YG4BQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.765478029 (597h52m29.636561074s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.765723596 (597h52m29.636806632s) [noise=true , address=145.79.227.16:4911 , piece_id=YMISM7TAOTUSZ6F3NUPFB5EHC4SCRZFBEKKEFPOG5SOWNJSFGVJA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.766612329 (597h52m29.637695364s) [noise=true , address=145.79.227.39:4015 , piece_id=3ATCT6C4I66WDJNPLZ2ADP7M5YBNFS6UMWJWIJAE4HSGCHSDDHKQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.767873170 (597h52m29.638956205s) [noise=true , address=145.79.227.2:4115 , piece_id=5LEYO73EBV4FUBEPCCOIOHQJBVXR4AQCZDZ3SFGBQLU7LVNTVHGQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.768854288 (597h52m29.639937333s) [noise=true , address=145.79.227.24:4583 , piece_id=CP4VKRHNS6C2GLXATFVG6QKTUMFZIIZQN56HIQZKXLC7Y7X4VK5A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.769467075 (597h52m29.64055011s) [noise=true , address=145.79.227.50:4374 , piece_id=J7SYIFAINVSDERN3UWZIIZFKEBSVWKIETCYFJS3W2BFI6VVQIMTA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.770806114 (597h52m29.64188915s) [noise=true , address=145.79.227.26:4653 , piece_id=ODX2XWJTGCNCA6TCCAJEWRCX2BJF2XKYT2EYE5UWGN2SR7OJFA5A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.771265229 (597h52m29.642348265s) [noise=true , address=145.79.227.19:4078 , piece_id=JRNJ6NQPPOQL42JE6P2ABL6JF5EZCZ74R4DYCKZ5DXVGY5BDLKGQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.772594400 (597h52m29.643677445s) [noise=true , address=77.90.31.11:4553 , piece_id=DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.780612017 (597h52m29.651695052s) [noise=true , address=45.91.116.13:4153 , piece_id=7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.783374417 (597h52m29.654457462s) [noise=true , address=145.79.227.41:4085 , piece_id=TPDBBLHTA43O7L4E325PSB2FC5ZYU3JETDE6HUYMYHY73NXOX4YA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.785132534 (597h52m29.656215579s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:31.472089583 (597h52m31.343172618s) [upload=524063]: Next returned nil segment Dec 30 13:21:31.472121153 (597h52m31.343204188s) [upload=524063]: Waiting for error group managing segments... Dec 30 13:21:31.752334330 (597h52m31.623417376s) [noise=true , address=145.79.227.19:4078 , piece_id=JRNJ6NQPPOQL42JE6P2ABL6JF5EZCZ74R4DYCKZ5DXVGY5BDLKGQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757288283 (597h52m31.628371328s) [noise=true , address=145.79.227.39:4015 , piece_id=3ATCT6C4I66WDJNPLZ2ADP7M5YBNFS6UMWJWIJAE4HSGCHSDDHKQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757559550 (597h52m31.628642595s) [noise=true , address=145.79.227.16:4911 , piece_id=YMISM7TAOTUSZ6F3NUPFB5EHC4SCRZFBEKKEFPOG5SOWNJSFGVJA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757889207 (597h52m31.628972242s) [noise=true , address=145.79.227.2:4115 , piece_id=5LEYO73EBV4FUBEPCCOIOHQJBVXR4AQCZDZ3SFGBQLU7LVNTVHGQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.761055006 (597h52m31.632138041s) [noise=true , address=145.79.227.11:4731 , piece_id=BFM7VAM5JIMZXXTGSOJGU4CR2BTS22DCDAS4U3DZ3VOJGUJJLOHA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.767159039 (597h52m31.638242074s) [noise=true , address=145.79.227.41:4085 , piece_id=TPDBBLHTA43O7L4E325PSB2FC5ZYU3JETDE6HUYMYHY73NXOX4YA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.771751394 (597h52m31.642834429s) [noise=true , address=145.79.227.42:4122 , piece_id=SZM3BNYSV3FRGT2E2WACH6DJCCT66NIHJELUT4WKWPOZKM2YG4BQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.789060603 (597h52m31.660143648s) [noise=true , address=145.79.227.24:4583 , piece_id=CP4VKRHNS6C2GLXATFVG6QKTUMFZIIZQN56HIQZKXLC7Y7X4VK5A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.790015191 (597h52m31.661098226s) [noise=true , address=145.79.227.50:4374 , piece_id=J7SYIFAINVSDERN3UWZIIZFKEBSVWKIETCYFJS3W2BFI6VVQIMTA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.853644600 (597h52m31.724727665s) [noise=true , address=145.79.227.26:4653 , piece_id=ODX2XWJTGCNCA6TCCAJEWRCX2BJF2XKYT2EYE5UWGN2SR7OJFA5A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.119951609 (597h52m31.991034654s) [noise=true , address=77.90.31.11:4553 , piece_id=DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.332028685 (597h52m32.20311173s) [noise=true , address=45.91.116.13:4153 , piece_id=7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.333908855 (597h52m32.20499189s) [noise=true , address=77.90.31.14:4661 , piece_id=YW4B33YWRWPXCFWRKOS57ATK5MXPRC2MLXFCCRBEAE4UH3S3H6FQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.333922632 (597h52m32.205005667s) [seg_pos={0 0}, upload=524063]: Segment reached optimal threshold of 13 pieces. Dec 30 13:21:32.334117362 (597h52m32.205200407s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334126250 (597h52m32.205209285s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334420179 (597h52m32.205503234s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334427603 (597h52m32.205510638s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334656860 (597h52m32.205739905s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334665336 (597h52m32.205748381s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334698228 (597h52m32.205781263s) [seg_pos={0 0}, upload=524063]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:21:32.334774003 (597h52m32.205857038s) [upload=524063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:21:32.334779222 (597h52m32.205862267s) [upload=524063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:21:32.625237599 (597h52m32.496320644s) [upload=524063]: Done uploading segments... Dec 30 13:21:32.625290089 (597h52m32.496373134s) [upload=524063]: Upload finished. err: <nil> Duration: 5.964388638s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X........X...........X..X........X...X...XXX.......X...X] duration: 30.001477943s trace: https://jaeger.dev.storj.io/trace/65483e5cb1af7d85 FAIL (2025-12-30 13:24:07 UTC) orbiter warning: context deadline exceeded Duration: 30.001502641s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XXX] duration: 5.001068201s trace: https://jaeger.dev.storj.io/trace/7ba56970f4e55d40 FAIL (2025-12-30 13:24:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001092215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X....X........X..............X.......X...X............] duration: 35.091657373s trace: https://jaeger.dev.storj.io/trace/327137fdf99c8cd2 FAIL (2025-12-30 13:23:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463267094 (597h44m10.334350129s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463636957 (597h44m10.334719992s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463733021 (597h44m10.334816076s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463967257 (597h44m10.335050292s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464317554 (597h44m10.335400589s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464332772 (597h44m10.335415817s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464747913 (597h44m10.335830958s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464994803 (597h44m10.336077848s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.465113149 (597h44m10.336196184s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.465730814 (597h44m10.336813859s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.466479270 (597h44m10.337562315s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.466845497 (597h44m10.337928542s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.467111013 (597h44m10.338194058s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.467663886 (597h44m10.338746921s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:11.913537776 (597h44m11.784620821s) [upload=523949]: Next returned nil segment Dec 30 13:13:11.913553607 (597h44m11.784636652s) [upload=523949]: Waiting for error group managing segments... Dec 30 13:13:12.310633846 (597h44m12.181716891s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.369636326 (597h44m12.240719361s) [noise=true , address=145.79.227.5:4248 , piece_id=GVI3MNAX2S2H6IKO67GTHYD6Z2FXPDPNRRSFEBRICJKOZYOP6BLA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.489444639 (597h44m12.360527674s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.552298983 (597h44m12.423382018s) [noise=true , address=64.49.13.27:4967 , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.637395539 (597h44m12.508478574s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.647923005 (597h44m12.51900604s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.718170647 (597h44m12.589253692s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.731374770 (597h44m12.602457815s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.792347862 (597h44m12.663430917s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.202419249 (597h44m13.073502284s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.235840199 (597h44m13.106923234s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.322780525 (597h44m13.193863571s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.442893078 (597h44m13.313976113s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.631615082 (597h44m13.502698127s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.631640270 (597h44m13.502723315s) [seg_pos={0 0}, upload=523949]: Segment reached optimal threshold of 14 pieces. Dec 30 13:13:13.631902971 (597h44m13.502986036s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF): protocol: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection Dec 30 13:13:13.631912879 (597h44m13.502995935s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632041715 (597h44m13.50312475s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 30 13:13:13.632057996 (597h44m13.503141031s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632602493 (597h44m13.503685528s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection Dec 30 13:13:13.632610257 (597h44m13.503693302s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632858089 (597h44m13.503941134s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 30 13:13:13.632864872 (597h44m13.503947907s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632959923 (597h44m13.504042958s) [seg_pos={0 0}, upload=523949]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:13:13.633117143 (597h44m13.504200188s) [upload=523949]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:13:13.633123475 (597h44m13.50420651s) [upload=523949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:13:13.920616664 (597h44m13.791699709s) [upload=523949]: Done uploading segments... Dec 30 13:13:13.920640600 (597h44m13.791723645s) [upload=523949]: Upload finished. err: <nil> Duration: 35.09167177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............X..X.X..X....X........XX................X.XX..] duration: 5.748644806s trace: https://jaeger.dev.storj.io/trace/1a1439816529e26e FAIL (2025-12-30 13:23:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA) context deadline exceeded ======LOGS====== Dec 30 13:21:29.510349015 (597h52m29.38143205s) [upload=524063]: Starting upload Dec 30 13:21:29.510459265 (597h52m29.3815423s) [upload=524063]: Done starting upload Dec 30 13:21:29.510558494 (597h52m29.381641529s) [upload=524063]: Uploading segments... Dec 30 13:21:29.510712408 (597h52m29.381795453s) [upload=524063]: Got next segment. Inline: false Dec 30 13:21:29.510754197 (597h52m29.381837242s) [upload=524063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 13:21:29.510759327 (597h52m29.381842362s) [upload=524063]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 13:21:29.762893408 (597h52m29.633976443s) [seg_pos={0 0}, upload=524063]: Begin upload segment... Dec 30 13:21:29.762954584 (597h52m29.634037629s) [seg_pos={0 0}, upload=524063]: Done begin upload segment. Dec 30 13:21:29.763018997 (597h52m29.634102032s) [noise=true , address=145.79.227.11:4731 , piece_id=BFM7VAM5JIMZXXTGSOJGU4CR2BTS22DCDAS4U3DZ3VOJGUJJLOHA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.763062069 (597h52m29.634145114s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.763987811 (597h52m29.635070856s) [noise=true , address=77.90.31.14:4661 , piece_id=YW4B33YWRWPXCFWRKOS57ATK5MXPRC2MLXFCCRBEAE4UH3S3H6FQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.764880712 (597h52m29.635963747s) [noise=true , address=145.79.227.42:4122 , piece_id=SZM3BNYSV3FRGT2E2WACH6DJCCT66NIHJELUT4WKWPOZKM2YG4BQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.765478029 (597h52m29.636561074s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.765723596 (597h52m29.636806632s) [noise=true , address=145.79.227.16:4911 , piece_id=YMISM7TAOTUSZ6F3NUPFB5EHC4SCRZFBEKKEFPOG5SOWNJSFGVJA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.766612329 (597h52m29.637695364s) [noise=true , address=145.79.227.39:4015 , piece_id=3ATCT6C4I66WDJNPLZ2ADP7M5YBNFS6UMWJWIJAE4HSGCHSDDHKQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.767873170 (597h52m29.638956205s) [noise=true , address=145.79.227.2:4115 , piece_id=5LEYO73EBV4FUBEPCCOIOHQJBVXR4AQCZDZ3SFGBQLU7LVNTVHGQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.768854288 (597h52m29.639937333s) [noise=true , address=145.79.227.24:4583 , piece_id=CP4VKRHNS6C2GLXATFVG6QKTUMFZIIZQN56HIQZKXLC7Y7X4VK5A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.769467075 (597h52m29.64055011s) [noise=true , address=145.79.227.50:4374 , piece_id=J7SYIFAINVSDERN3UWZIIZFKEBSVWKIETCYFJS3W2BFI6VVQIMTA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.770806114 (597h52m29.64188915s) [noise=true , address=145.79.227.26:4653 , piece_id=ODX2XWJTGCNCA6TCCAJEWRCX2BJF2XKYT2EYE5UWGN2SR7OJFA5A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.771265229 (597h52m29.642348265s) [noise=true , address=145.79.227.19:4078 , piece_id=JRNJ6NQPPOQL42JE6P2ABL6JF5EZCZ74R4DYCKZ5DXVGY5BDLKGQ, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.772594400 (597h52m29.643677445s) [noise=true , address=77.90.31.11:4553 , piece_id=DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.780612017 (597h52m29.651695052s) [noise=true , address=45.91.116.13:4153 , piece_id=7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.783374417 (597h52m29.654457462s) [noise=true , address=145.79.227.41:4085 , piece_id=TPDBBLHTA43O7L4E325PSB2FC5ZYU3JETDE6HUYMYHY73NXOX4YA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:29.785132534 (597h52m29.656215579s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Uploading piece... Dec 30 13:21:31.472089583 (597h52m31.343172618s) [upload=524063]: Next returned nil segment Dec 30 13:21:31.472121153 (597h52m31.343204188s) [upload=524063]: Waiting for error group managing segments... Dec 30 13:21:31.752334330 (597h52m31.623417376s) [noise=true , address=145.79.227.19:4078 , piece_id=JRNJ6NQPPOQL42JE6P2ABL6JF5EZCZ74R4DYCKZ5DXVGY5BDLKGQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757288283 (597h52m31.628371328s) [noise=true , address=145.79.227.39:4015 , piece_id=3ATCT6C4I66WDJNPLZ2ADP7M5YBNFS6UMWJWIJAE4HSGCHSDDHKQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757559550 (597h52m31.628642595s) [noise=true , address=145.79.227.16:4911 , piece_id=YMISM7TAOTUSZ6F3NUPFB5EHC4SCRZFBEKKEFPOG5SOWNJSFGVJA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.757889207 (597h52m31.628972242s) [noise=true , address=145.79.227.2:4115 , piece_id=5LEYO73EBV4FUBEPCCOIOHQJBVXR4AQCZDZ3SFGBQLU7LVNTVHGQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.761055006 (597h52m31.632138041s) [noise=true , address=145.79.227.11:4731 , piece_id=BFM7VAM5JIMZXXTGSOJGU4CR2BTS22DCDAS4U3DZ3VOJGUJJLOHA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.767159039 (597h52m31.638242074s) [noise=true , address=145.79.227.41:4085 , piece_id=TPDBBLHTA43O7L4E325PSB2FC5ZYU3JETDE6HUYMYHY73NXOX4YA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.771751394 (597h52m31.642834429s) [noise=true , address=145.79.227.42:4122 , piece_id=SZM3BNYSV3FRGT2E2WACH6DJCCT66NIHJELUT4WKWPOZKM2YG4BQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.789060603 (597h52m31.660143648s) [noise=true , address=145.79.227.24:4583 , piece_id=CP4VKRHNS6C2GLXATFVG6QKTUMFZIIZQN56HIQZKXLC7Y7X4VK5A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.790015191 (597h52m31.661098226s) [noise=true , address=145.79.227.50:4374 , piece_id=J7SYIFAINVSDERN3UWZIIZFKEBSVWKIETCYFJS3W2BFI6VVQIMTA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:31.853644600 (597h52m31.724727665s) [noise=true , address=145.79.227.26:4653 , piece_id=ODX2XWJTGCNCA6TCCAJEWRCX2BJF2XKYT2EYE5UWGN2SR7OJFA5A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.119951609 (597h52m31.991034654s) [noise=true , address=77.90.31.11:4553 , piece_id=DXU7YVDGZ2VHVB733OFL5XFHQKTCX2R7PF4KLULFTFHAD7DVJIDA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.332028685 (597h52m32.20311173s) [noise=true , address=45.91.116.13:4153 , piece_id=7J4Z3PRNVFERHSNMZRVGQ63UVMZM26KM2CZSZJT2SNAVAPZDCK4A, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.333908855 (597h52m32.20499189s) [noise=true , address=77.90.31.14:4661 , piece_id=YW4B33YWRWPXCFWRKOS57ATK5MXPRC2MLXFCCRBEAE4UH3S3H6FQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: <nil> Dec 30 13:21:32.333922632 (597h52m32.205005667s) [seg_pos={0 0}, upload=524063]: Segment reached optimal threshold of 13 pieces. Dec 30 13:21:32.334117362 (597h52m32.205200407s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334126250 (597h52m32.205209285s) [noise=true , address=45.91.116.9:4261 , piece_id=BGTLCOS2EFUMGI5NEAONMWBV22XQWSKYTXIKPV4PP36YKZVFG4MQ, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334420179 (597h52m32.205503234s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334427603 (597h52m32.205510638s) [noise=true , address=77.90.31.15:4697 , piece_id=MNBIYAN2WN5URT2TFZ42W6TVDWJ2C2M3JZS7CV2KIONMDWR4MEBA, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334656860 (597h52m32.205739905s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 13:21:32.334665336 (597h52m32.205748381s) [noise=true , address=145.79.227.44:4193 , piece_id=BRYLQPJTJ5R6EC6BEUP7SRNXHYQQBQEEUFFJ6LYHE5JAIFEJIXAQ, seg_pos={0 0}, upload=524063]: Long tail cancellation due to optimal threshold Dec 30 13:21:32.334698228 (597h52m32.205781263s) [seg_pos={0 0}, upload=524063]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 13:21:32.334774003 (597h52m32.205857038s) [upload=524063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:21:32.334779222 (597h52m32.205862267s) [upload=524063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:21:32.625237599 (597h52m32.496320644s) [upload=524063]: Done uploading segments... Dec 30 13:21:32.625290089 (597h52m32.496373134s) [upload=524063]: Upload finished. err: <nil> Duration: 5.748679844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXXXXXXXX.X.XX] duration: 5.000555112s trace: https://jaeger.dev.storj.io/trace/308fdd731f62a63d FAIL (2025-12-30 13:23:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000576512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X....X........X..............X.......X...X...........] duration: 51.853561817s trace: https://jaeger.dev.storj.io/trace/327137fdf99c8cd2 FAIL (2025-12-30 13:23:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463267094 (597h44m10.334350129s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463636957 (597h44m10.334719992s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463733021 (597h44m10.334816076s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.463967257 (597h44m10.335050292s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464317554 (597h44m10.335400589s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464332772 (597h44m10.335415817s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464747913 (597h44m10.335830958s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.464994803 (597h44m10.336077848s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.465113149 (597h44m10.336196184s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.465730814 (597h44m10.336813859s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.466479270 (597h44m10.337562315s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.466845497 (597h44m10.337928542s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.467111013 (597h44m10.338194058s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:10.467663886 (597h44m10.338746921s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Uploading piece... Dec 30 13:13:11.913537776 (597h44m11.784620821s) [upload=523949]: Next returned nil segment Dec 30 13:13:11.913553607 (597h44m11.784636652s) [upload=523949]: Waiting for error group managing segments... Dec 30 13:13:12.310633846 (597h44m12.181716891s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.369636326 (597h44m12.240719361s) [noise=true , address=145.79.227.5:4248 , piece_id=GVI3MNAX2S2H6IKO67GTHYD6Z2FXPDPNRRSFEBRICJKOZYOP6BLA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.489444639 (597h44m12.360527674s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.552298983 (597h44m12.423382018s) [noise=true , address=64.49.13.27:4967 , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.637395539 (597h44m12.508478574s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.647923005 (597h44m12.51900604s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.718170647 (597h44m12.589253692s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.731374770 (597h44m12.602457815s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:12.792347862 (597h44m12.663430917s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.202419249 (597h44m13.073502284s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.235840199 (597h44m13.106923234s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.322780525 (597h44m13.193863571s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.442893078 (597h44m13.313976113s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.631615082 (597h44m13.502698127s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil> Dec 30 13:13:13.631640270 (597h44m13.502723315s) [seg_pos={0 0}, upload=523949]: Segment reached optimal threshold of 14 pieces. Dec 30 13:13:13.631902971 (597h44m13.502986036s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF): protocol: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection Dec 30 13:13:13.631912879 (597h44m13.502995935s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632041715 (597h44m13.50312475s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 30 13:13:13.632057996 (597h44m13.503141031s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632602493 (597h44m13.503685528s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection Dec 30 13:13:13.632610257 (597h44m13.503693302s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632858089 (597h44m13.503941134s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 30 13:13:13.632864872 (597h44m13.503947907s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold Dec 30 13:13:13.632959923 (597h44m13.504042958s) [seg_pos={0 0}, upload=523949]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 13:13:13.633117143 (597h44m13.504200188s) [upload=523949]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 13:13:13.633123475 (597h44m13.50420651s) [upload=523949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 13:13:13.920616664 (597h44m13.791699709s) [upload=523949]: Done uploading segments... Dec 30 13:13:13.920640600 (597h44m13.791723645s) [upload=523949]: Upload finished. err: <nil> Duration: 51.853582467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 16.666312832s trace: https://jaeger.dev.storj.io/trace/7b47a373389cd7c3 FAIL (2025-12-30 13:23:21 UTC) orbiter warning: context deadline exceeded Duration: 16.666362726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X........X...........X..X........X...X...XXX.......X...] duration: 17.870117907s trace: https://jaeger.dev.storj.io/trace/19af6f5455003ec6 FAIL (2025-12-30 13:23:19 UTC) orbiter warning: context deadline exceeded Duration: 17.870148294s edge gateway us-select-1

1
...
11243
11244
11245
11246
11247
...
18141